Celery & Async Calculations
When a calculation triggers other calculations — say, a parent that kicks off five children — the framework can dispatch those children to Celery workers in parallel instead of processing them one by one. This is transparent to your code: you write the same calculate() method either way.
Architecture
flowchart LR
A["User clicks
Calculate"] --> B["Django
(main process)"]
B --> C{"CELERY_ACTIVE
= true?"}
C -- Yes --> D["Redis / Memurai
(message broker)"]
D --> E["Celery Worker 1"]
D --> F["Celery Worker 2"]
D --> G["Celery Worker N"]
C -- No --> H["Synchronous
fallback"]
E --> I["Results
(PostgreSQL backend)"]
F --> I
G --> I
H --> IThe key design principle: Celery is optional. If the broker is down or Celery isn't configured, the framework falls back to synchronous processing automatically. Your calculations still run — just sequentially instead of in parallel.
Prerequisites: Redis (or Memurai on Windows)
Celery needs a message broker to dispatch tasks. Lex App uses Redis by default.
Linux / macOS — Install Redis
# Ubuntu / Debian sudo apt install redis-server sudo systemctl enable --now redis # macOS (Homebrew) brew install redis brew services start redisVerify with
redis-cli ping— you should getPONG.
Windows — Install Memurai
Redis doesn't run natively on Windows. Use Memurai instead — a Windows-native, Redis-compatible in-memory datastore.
- Download the Developer Edition (free, intended for development and testing) from memurai.com/get-memurai
- Run the installer — Memurai starts automatically as a Windows service
- Verify with
memurai-cli ping— you should getPONGMemurai is fully compatible with the Redis API, so no code changes are needed. It listens on the same default port (
6379).Limitations of the Developer Edition: restarts required every 10 days, max 10 connected hosts, 50% RAM cap. For production on Windows, use the Enterprise Edition.
By default, the framework connects to redis://127.0.0.1:6379/1 for local development (both Linux/macOS Redis and Windows Memurai).
How It Works
The @lex_shared_task Decorator
To make a calculate() method Celery-capable, decorate it with @lex_shared_task:
from lex.lex_app.celery_tasks import lex_shared_task
class HeavyReport(CalculationModel):
@lex_shared_task
def calculate(self):
# Your calculation logic — same as always
...@lex_shared_task wraps your method into an enhanced Celery task with:
- Context-aware dispatch — respects
WaitForTasksandFireAndForgetcontext managers - Automatic status callbacks — the
CallbackTaskbase class updatesis_calculatedtoSUCCESSorERRORon completion - Context propagation — calculation IDs and audit logging context are forwarded to workers, including the active child model during batch dispatch
Without the decorator, calculate() always runs synchronously — even when CELERY_ACTIVE=true.
The Dispatch Flow
When a user clicks Calculate ▶️, the framework decides sync vs. async:
- Is
CELERY_ACTIVE=trueset in the environment? - Does the
calculate()method have a.delay()attribute (i.e., is it decorated with@lex_shared_task)?
If both are true, the calculation is dispatched to a Celery worker via calc_and_save.delay(). Otherwise, it runs synchronously in the request thread.
For batch calculations (a parent triggering children), the framework uses CeleryTaskDispatcher to:
- Group the child models into batches (clustered by calculation order)
- Dispatch each group as a separate
calc_and_saveCelery task inside aWaitForTaskscontext - Monitor task completion (blocks until all groups finish)
- Retry failed groups synchronously as a fallback
# You don't call any of this directly.
# The framework handles dispatch when your calculate() triggers children.
class ParentCalculation(CalculationModel):
@lex_shared_task
def calculate(self):
children = ChildCalculation.objects.filter(quarter=self.quarter)
for child in children:
child.is_calculated = "IN_PROGRESS"
child.save() # Triggers child calculation → dispatched to CeleryEnvironment Variables
Variable | Where to Set | Purpose |
|---|---|---|
|
| Enables the Celery dispatch path. Without this, all calculations run synchronously. |
| Worker command only | Tells the framework the process is a Celery worker (skips app startup tasks like data loading). Do not set this in the main app's |
Add CELERY_ACTIVE=true to your project's .env file so it's always active when you run the app (from the terminal or PyCharm):
CELERY_ACTIVE=trueRunning Workers
Start Celery workers in a separate terminal alongside your running app:
Linux / macOS
IS_RUNNING_IN_CELERY=true CELERY_ACTIVE=true lex celery -A lex_app worker \ --loglevel=info \ --concurrency=12 \ --prefetch-multiplier=1 \ -n worker1@%h
Flag
Meaning
--concurrency=12Number of parallel worker threads/processes
--prefetch-multiplier=1Don't prefetch extra tasks — important for long-running calculations
-n worker1@%hWorker name (
%hexpands to hostname). Useworker2@%h,worker3@%h, etc. for additional workersYou can run multiple worker processes on the same machine by changing the
-nname.
On Windows, Celery's default prefork pool isn't supported. Use the
--pool=soloor--pool=threadsflag, or run workers via WSL.
For development, you can skip running workers entirely — everything runs synchronously by default when CELERY_ACTIVE is not set or false.
WaitForTasks and FireAndForget
The framework provides two context managers for advanced dispatch control. You typically don't need these — the framework uses them internally — but they're available for custom task orchestration.
Legacy names:
RunInCelery,AwaitDispatchandUnblockCelerystill work as aliases forWaitForTasks/FireAndForget. They have no removal date set, but new code should prefer the canonical names — you'll get a deprecation warning before they're removed.
WaitForTasks
Dispatches @lex_shared_task-decorated calls to Celery workers and blocks on exit until every dispatched task has finished. Without this context, tasks run synchronously in the current thread.
from lex.lex_app.celery_tasks import WaitForTasks
with WaitForTasks():
my_task(data) # dispatched to a Celery worker
other_task(data) # dispatched to a Celery worker (runs in parallel)
# Execution reaches here only after BOTH tasks have completedSelective dispatch
You can control which tasks get dispatched and which stay synchronous:
# Only dispatch calc_and_save — everything else runs synchronously
with WaitForTasks(include_tasks={"calc_and_save"}):
...
# Dispatch everything EXCEPT initial_data_upload
with WaitForTasks(exclude_tasks={"initial_data_upload"}):
...Nesting
WaitForTasks blocks can be nested. Each scope independently tracks and waits for only the tasks dispatched within it:
with WaitForTasks(): # outer scope
compute_portfolio.delay(fund_a) # dispatched, tracked by outer
compute_portfolio.delay(fund_b) # dispatched, tracked by outer
with WaitForTasks(): # inner scope
compute_nav.delay(q1) # dispatched, tracked by inner
compute_nav.delay(q2) # dispatched, tracked by inner
# ← blocks here until q1 and q2 finish
generate_report.delay(fund_a) # dispatched, tracked by outer
# ← blocks here until fund_a, fund_b, and the report finishAll four portfolio/NAV tasks run in parallel on Celery workers, but the calling thread waits at each scope boundary for the tasks it owns.
FireAndForget
Dispatches tasks to Celery without waiting for them. Use this for side-effects that don't affect the caller's correctness — notifications, cache warming, analytics, etc.
On its own, FireAndForget simply dispatches and moves on:
from lex.lex_app.celery_tasks import FireAndForget
with FireAndForget():
send_report_email(report) # dispatched, nobody waits
notify_slack_channel(report) # dispatched, nobody waits
# Execution continues immediately — emails may still be sendingWhen nested inside a WaitForTasks block, it overrides the blocking behaviour for specific calls:
from lex.lex_app.celery_tasks import WaitForTasks, FireAndForget
with WaitForTasks():
compute_nav.delay(q1) # dispatched, parent WILL wait
with FireAndForget():
send_report_email(report) # dispatched, parent WON'T wait
notify_slack_channel(report) # dispatched, parent WON'T wait
compute_nav.delay(q2) # dispatched, parent WILL wait
# Blocks until q1 and q2 finish. Emails/Slack may still be in flight.When to use FireAndForget
Use case | Why fire-and-forget |
|---|---|
Email / Slack / webhook notifications | Caller doesn't need the result |
Audit-log enrichment (async) | Nice-to-have, not on the critical path |
Cache warming / precomputation | Optimisation, not required for correctness |
Analytics / telemetry events | Tracking shouldn't slow calculations |
The key question: "If this task fails or is delayed, does the caller break?" If no → FireAndForget. If yes → let WaitForTasks handle it.
Priority hierarchy
When contexts are nested, the innermost matching context wins:
Priority | Context | Behaviour |
|---|---|---|
1 (highest) |
| Dispatch to Celery, don't wait |
2 |
| Dispatch to Celery, block on exit |
3 (default) | No context | Run synchronously in-process |
Failure Handling
The dispatcher handles failures at multiple levels:
Failure Type | What Happens |
|---|---|
Celery import fails | Entire batch runs synchronously |
Single task dispatch fails | That group runs synchronously, others continue on Celery |
Task execution fails | Failed group retried synchronously |
Broker goes down mid-run | Remaining groups run synchronously |
This means your calculations are resilient — they always complete, even if the infrastructure has issues.
When to Use Celery
Scenario | Celery Useful? |
|---|---|
Single calculation, no children | No — no parallelism to gain |
Parent triggers 2–3 children | Maybe — overhead may not be worth it |
Parent triggers 10+ children | Yes — significant speedup |
Long-running calculations (minutes+) | Yes — prevents blocking the web process |
Development / small projects | No — synchronous is simpler |
Monitoring
Celery tasks are logged with full context:
Starting Celery dispatch for 5 groups containing 23 total models
Dispatch summary: 5/5 groups dispatched to Celery
Task processing completed: 5/5 tasks successfulYou can also monitor Celery with Flower:
lex celery -A lex_app flowerThis gives you a web dashboard at http://localhost:5555 with real-time task monitoring.
Was this helpful?
Your feedback shapes what we write next.
