PHP → Go

PHP Queue Workers Are Often a Better Go Migration Target Than Your API

A background consumer has no latency contract, an existing retry safety net and a one-page interface. That makes it the lowest-risk place to find out whether Go actually helps you.

When a team decides to try Go, the instinct is to pick an HTTP endpoint. It is visible, it has a latency number attached, and it feels like the real system. It is also the worst possible place to start, because a live endpoint has a user waiting on the other end and a cutover you cannot undo silently. Moving PHP workers to Go is a better opening move on every axis that matters: the interface is a message contract rather than a request/response surface, the queue already gives you a retry mechanism, nothing user-facing breaks if the first version is wrong, and you can run both implementations against the same stream and compare their output directly.

If you have been searching for PHP queue workers Go comparisons, you have probably already found benchmark posts. Those are the least interesting part. What decides whether this migration pays off is memory per unit of concurrency, process lifetime, and how carefully you handle the semantics your broker actually provides.

Why a worker is a safer first target than an API endpoint

Four properties, and each one removes a category of risk.

No latency contract during cutover. A background job that takes 400 ms instead of 90 ms inconveniences nobody as long as throughput keeps up with arrival rate. You can deploy a first version that is merely correct, measure it, and improve it. An API endpoint that regresses at p99 is an incident.

An existing safety net. Queues already assume failure. If your Go consumer panics on message 4,000, the message is redelivered, to the PHP consumer, if you have kept it running. There is no equivalent of “redeliver this HTTP request to the old implementation” once a response has gone out.

A narrow interface. The contract is one message schema. Not a controller, a middleware stack, a serializer, an auth layer and a response envelope. You can write it down in a few dozen lines of struct definitions and validate both implementations against the same fixtures.

Trivial equivalence testing. You can point both consumers at copies of the same stream and diff their side effects. Genuine A/B comparison on production data, before anything depends on the new code.

What job processing actually costs

Before writing any Go, profile the jobs themselves, because the answer determines whether this is worth doing. Split each job type’s wall time into database, external HTTP, CPU in your own code, and framework bootstrap.

If a job is 90% waiting on PostgreSQL and a payment API, Go will not make it finish sooner. It will let you run far more of them concurrently in the same memory footprint, which is a throughput and cost argument rather than a latency one, still valid, but you should know which claim you are making. If a job is genuinely CPU-bound in your own code, parsing large CSV or XML, image and PDF work, cryptography, compression, geometry, large in-memory transformations, the runtime difference is direct.

Framework bootstrap deserves separate attention. A PHP worker that boots a container, registers providers, and builds an event dispatcher per job pays that cost every single message. Long-running worker daemons (Laravel’s queue:work, Symfony’s messenger:consume, or RoadRunner) already amortise it, and if you are still on a per-job bootstrap model, switching to a resident PHP worker is a much cheaper experiment than rewriting in Go. Try it first.

Memory: N processes versus one process with N goroutines

This is usually the decisive technical argument, and it is worth being precise rather than triumphant about it.

A PHP worker is a process. Concurrency 40 means 40 processes, each with its own copy of the interpreter state, the framework’s object graph, and whatever the current job allocated. Suppose, as an illustrative figure, each worker settles at 120 MB resident. Forty workers is roughly 4.8 GB before your jobs do anything interesting, and each of those workers holds its own database connection, so your PostgreSQL max_connections becomes a planning constraint too. Scaling concurrency scales memory linearly, and on Kubernetes that means the pod’s memory request grows with every increment of parallelism.

A Go consumer is one process. Goroutines start at a couple of kilobytes of stack and grow on demand, so tens of thousands of in-flight I/O-bound jobs are unremarkable. The interpreter state and framework graph exist once. Connections come from a shared database/sql pool with SetMaxOpenConns bounding what the database sees, independent of how many jobs are in flight. That decoupling of concurrency from both memory and connection count is the property people are actually buying.

That last row is the honest cost, and it leads directly to the next section.

Why PHP workers get restarted after N jobs

Long-lived PHP processes accumulate state: static properties, ORM identity maps and unit-of-work registries, in-memory caches, registered event listeners, container instances holding request-scoped data, and connections that went stale while idle. Beyond memory growth, this causes correctness bugs, an entity cached in the identity map from job 300 is returned to job 900 after the row has changed underneath it.

The community answer is process recycling: --max-jobs, --memory=128, pm.max_requests. It works, and it is a legitimate engineering choice, bounded process lifetime converts a class of leaks into a non-issue. But it means paying bootstrap repeatedly, and it means you never really solved the leak.

Go removes the excuse. A Go consumer runs for weeks, so a slowly growing map, an unclosed response body, a goroutine leaked per message, or a context never cancelled will eventually take the process down. You now need pprof heap and goroutine profiles exposed and occasionally read, defer resp.Body.Close() as reflex, and per-job contexts with timeouts. Bounded lifetime is replaced by discipline. That is a fair trade, but it is a trade.

Concurrency models

In PHP the knob is process count, tuned by a supervisor. Parallelism inside a single job is awkward, real threads require ext-parallel, and curl_multi or Fibers cover narrower cases than people hope. Backpressure is implicit: a process can only hold one job, so it cannot over-fetch.

In Go the knob is a bounded worker pool: a buffered channel of messages, a fixed number of goroutines ranging over it, and a sync.WaitGroup for shutdown. This is where the most common mistake happens, spawning go handle(msg) per message with no bound. It works in staging and then, on a backlog of 200,000 messages, opens 200,000 concurrent goroutines that exhaust your database pool, your file descriptors and the downstream API’s rate limit at the same time. Backpressure that PHP gave you for free is now something you must implement deliberately, and the mechanism is simply: do not fetch more than you have capacity to process. Prefetch limits, semaphores and channel buffer sizes are your flow control, and they should be sized against the downstream constraint, typically database connections or an external API quota, not against CPU count.

Broker semantics differ more than the client libraries suggest

Every broker exposes “receive a message, do work, acknowledge”. The delivery and redelivery mechanics underneath are quite different, and getting them wrong is how a migration produces duplicate charges.

RabbitMQ

Consumers are pushed messages up to a prefetch limit (basic.qos), and each message is explicitly acknowledged, rejected, or negatively acknowledged with requeue. Prefetch is your primary tuning and backpressure knob: too high and one consumer hoards a backlog while others idle and redelivery on crash becomes large; too low and you add a round trip per message. There is no server-side delay on requeue, so exponential backoff needs either a dead-letter exchange with a per-queue TTL or a delayed-message plugin, requeueing immediately in a loop is how you build a hot spin on a poison message. Note that in Go you must not share a channel across goroutines for publishing, and connection recovery is not automatic in the way some PHP libraries hide it.

Kafka

Not a queue. A partitioned log with consumer groups, where ordering is guaranteed per partition and parallelism is capped by partition count, twenty goroutines against four partitions gives you four-way parallelism. Progress is a committed offset, not a per-message ack, which means processing concurrently within a partition breaks the offset model unless you track completion carefully. Rebalances happen when members join, leave, or exceed the poll interval, so a slow handler can trigger a rebalance and cause redelivery. Retrying by blocking on one message halts the whole partition; the usual pattern is to publish failures to a retry topic and move on.

SQS-style

Messages are pulled, then hidden for a visibility timeout rather than locked. If your handler outruns the visibility timeout, the message becomes visible again and a second worker starts processing it while the first is still running, the classic source of duplicates. Long-running handlers must extend visibility periodically as a heartbeat. Redrive to a dead-letter queue after a maximum receive count is built in, and FIFO queues trade throughput for ordering by message group.

The reason to spell this out: whichever broker you use, its semantics are at-least-once. Which brings us to the part that matters more than the language.

Idempotency is the actual contract

Any consumer that can be redelivered a message must produce the same end state when it processes that message twice. This is true of your PHP worker today; the difference is that during a side-by-side migration you are deliberately increasing the chance of duplicate delivery, so latent non-idempotency will surface.

Practical mechanisms, in rough order of preference:

• Natural idempotency. UPDATE ... SET status = 'shipped' WHERE id = ? is safe to repeat. Prefer designs that are naturally repeatable.
• A unique constraint. Insert with a deterministic key derived from the message and treat a unique violation as success. The database enforces it; no application logic to get wrong.
• A processed-messages table. Insert (message_id, handler) in the same transaction as the effect, with a primary key on the pair. If the insert conflicts, skip. This works only if the effect is in the same database as the ledger.
• Idempotency keys for external calls. Pass a stable key on every outbound request that creates something, payment providers support this specifically because at-least-once systems are everywhere.

Two things to avoid: a SELECT followed by an INSERT outside a transaction, which is a race with an unlucky window, and a Redis SETNX marker set before the work completes, which silently drops a job if the process dies mid-flight.

Retries, backoff, poison messages, dead-letter queues

Distinguish retryable from terminal failures at the handler level and return them differently. A 503 from a provider, a lock timeout, a deadlock, retry. A validation failure, a missing referenced entity, a JSON payload that does not match the schema, will never succeed, so retrying it wastes capacity and delays real work.

For retryable failures, use exponential backoff with full jitter and a cap. Without jitter, a provider outage produces synchronised retry waves that hit the moment it recovers. Set a maximum attempt count and route exhausted messages to a dead-letter queue.

Then treat the DLQ as an operational surface rather than a graveyard: alert on non-zero depth and on arrival rate, keep the original payload plus the failure reason and attempt count, and build a replay path you have actually tested. A DLQ nobody looks at is a silent data-loss channel with extra steps.

Poison messages deserve specific handling. An unparseable message will fail identically forever; detect malformed payloads at the boundary and dead-letter them on the first attempt rather than burning the retry budget. During migration this matters twice over, because a schema mismatch between what the PHP producer emits and what the Go consumer expects is the most likely early bug you will hit.

Observability: tracing a job end to end

You cannot verify a migration you cannot observe, so this comes before the cutover, not after.

Emit structured JSON logs with a consistent field set, message_id, job_type, attempt, trace_id, duration_ms, outcome, from both implementations, using the same field names, so a single query compares them. Propagate traceparent from the producer into the message headers and continue the span in the consumer; a job trace that starts at the HTTP request that enqueued it is how you diagnose the interesting failures. Export metrics for jobs processed by type and outcome, handler duration histograms, in-flight count, retry count, DLQ depth, and consumer lag or queue age. Queue age, the time the oldest unprocessed message has been waiting, is the alert that best predicts a customer noticing.

Graceful shutdown

Consumers are killed constantly: deploys, autoscaling, node drains, spot reclamation. Handling that correctly is not optional, and it is one of the places where a Go rewrite is genuinely easy to get wrong.

The sequence: catch SIGTERM with signal.NotifyContext, immediately stop fetching new messages, let in-flight handlers finish under a deadline, acknowledge or explicitly requeue whatever completed, close the broker connection and drain the database pool, then exit. Kubernetes sends SIGTERM and waits terminationGracePeriodSeconds (30 by default) before SIGKILL, so your drain deadline must be shorter than that value, if your longest job takes 90 seconds, either raise the grace period or make the job resumable. Jobs killed mid-flight are not lost, because they are redelivered, but that only holds if the handler is idempotent. The two requirements are linked.

PHP’s equivalents (pcntl_signal handling in queue:work, checking a stop flag between jobs) work well; the point is that whatever your current workers do about shutdown, the Go version must do at least as well before it takes real traffic.

Rollout: run both consumers side by side

The migration itself is where the risk actually lives. Run both, shift gradually, compare continuously.

        producer (PHP monolith)
                 |
        [ orders.events topic ]
                 |
     +-----------+-----------+
     |                       |
 PHP consumer           Go consumer
   90% share              10% share
     |                       |
     +-----------+-----------+
                 |
        [ same database ]
                 |
      compare: outcome counts,
      duration, retries, DLQ,
      row-level diff on samples

  rollback = set Go share to 0%
             (config change, no deploy)

How you split depends on the broker. With RabbitMQ, bind a second queue to the exchange and route by a header or key hash so a deterministic slice of messages goes to the Go consumer. With Kafka, use separate consumer groups, either give the Go group a subset of partitions, or run it in shadow mode where it processes everything and writes to a comparison table instead of the real one. With SQS, a small router that forwards a percentage to a second queue does the job.

Shadow mode first, if the jobs allow it: the Go consumer reads real messages, performs all the work, and writes its results somewhere inert. You get equivalence data with zero production risk. It is only possible for jobs whose side effects you can redirect, which is another reason to pick the first target carefully, the general principles of routing a slice of traffic to new code while the old path stays live apply directly here.

Verifying equivalence

Define what “the same” means before you start, and check it mechanically: outcome counts per job type over the same window, row-level diffs on records written by each implementation, count and content of outbound side effects such as emails or webhooks, error rate and error type distribution, duration percentiles, and DLQ arrivals per implementation. Run at 1%, then 10%, then 50%, holding each step long enough to cover a full daily cycle, the batch job that runs at 03:00 has different characteristics from midday traffic, and a percentage rollout that only ran during business hours has not been tested.

Rollback

Keep the PHP consumer deployed and running the whole time, not commented out and not deleted. Rollback should be a configuration change that takes effect without a deploy: set the Go share to zero, and messages flow to the PHP consumer that has been running all along. Write the abort thresholds down in advance, error rate above X, DLQ arrivals above Y, queue age above Z, so the decision to roll back is not a debate at 02:00. Only remove the PHP path after the Go consumer has held 100% for a few weeks including a full billing or reporting cycle.

When not to move PHP workers to Go

A short list of situations where I would tell you to keep the PHP workers. Job volume is low and the workers are not a cost or latency problem, there is nothing to recover. The jobs are thin wrappers around framework features (mail with Blade or Twig templates, ORM-heavy domain logic, notification stacks) so porting means reimplementing framework behaviour, which is a large surface for a small gain. The handlers are almost entirely waiting on a rate-limited external API, which caps throughput regardless of runtime. Nobody on the team writes Go, and this consumer would become the only Go service in production with no second maintainer. Or the real problem is a missing index on the table every job updates, in which case start with finding out what is actually slow before changing language.

Deciding

The useful framing is not “PHP or Go” but “which specific workloads have a shape that a process-per-unit-of-work runtime handles badly”. Usually that is a small number of job types, high-volume, CPU-bound, or needing concurrency well beyond what per-process memory allows, and everything else is fine where it is. The runtime-level differences that make each language suit different work are the background; your profiler decides which category each job falls into.

If you want that decision made with evidence rather than preference, a scoped readiness audit covering job profiles, memory per worker, broker semantics and a concrete rollout plan is the efficient way to get it, and an assessment of what to keep in PHP versus extract into Go will sometimes conclude that a resident PHP worker with a fixed prefetch and three new indexes is the whole answer. When Go is the right call, a single queue consumer is the cheapest way to prove it, small enough to finish, isolated enough to roll back, and honest enough to measure.