Driving real iPhones on ALI Remote

Every phone you touch here is a physical handset on a rack, usually signed in to a real account that earns someone money. There is no sandbox and no undo. A tap that lands on the wrong thing is a real action on a real profile.

Read the two rules first. Everything else is detail.

Rule 1: an acknowledgement is not an effect. A 200 means the command reached the phone, not that the screen changed. Gestures say so explicitly, with effect_observed: false. Take a screenshot and look before you act on the assumption that something worked.

Rule 2: never loop on a failure. Two attempts is the whole strategy. A phone that refuses twice will refuse a thousand times, and the loop is what turns a stuck phone into a burnt account, a spent credit balance, or both.

Which door to use

They are the same engine and the same limits, and they draw on the same purchased balance. Do not mix them in one flow: you will double-spend credits and confuse yourself about which credential did what.

Get the reference with the API's own documents, which are generated from the code that validates requests and cannot drift. On your ALI Remote origin, today https://beta.aliremote.com, they are at /docs/api (or /docs/api.md), /docs/mcp, /llms.txt and /api/v1/openapi.json.

Orientation: read before you drive

1. List the phones. list_devices / GET /v1/devices. A phone is addressable by its opaque public_id (ph_…) or by the sticker on the back (inventory_id), if the rack is labelled.
2. Check it is online AND casting. Online means the box can see it. Casting means the screen is being mirrored, and without it you are driving blind. An action against a phone that is not casting is refused rather than silently accepted.
3. Take a screenshot. This is not optional. It is where your coordinates come from and it is the only way you know what is on screen right now.

capture_screen(device) -> image

Coordinates you read off that image are the ones you send back, unscaled. Do not multiply by a device pixel ratio. Do not reuse coordinates from a screenshot you took five minutes ago.

The one thing that surprises everyone: two kinds of verb

Verbs split into two tiers with completely different reliability, and knowing which is which saves days.

Pointer verbs work on any casting phone. They are driven through the handset's cursor:

tap · swipe · swipe_direction · scroll · drag · hold_drag · press (home, app switcher, lock, enter, arrows, volume…) · app_close · capture_screen · calibrate · recast

Helper verbs go through a small helper installed on the phone itself. They are slower (several seconds is normal) and they can fail on a phone where every pointer verb works perfectly:

text (beyond short plain ASCII) · clipboard_get · clipboard_set · open_url · wifi · cellular · airplane · torch · brightness · device_ip

Rotating the IP is its own endpoint rather than an action, because it is a half-minute cycle: POST /v1/devices/{id}/ip-rotation starts it and answers 202, then poll GET on the same path until status leaves running. rotated carries ip_after and a fraud_score (0 clean to 100, fraud_risk low / medium / high / critical); unchanged means the carrier gave the same address twice, so wait a minute and try again. A phone on ALI Wi-Fi answers 409 rotation_unavailable and never will rotate; that is the network, not the phone. device_ip with public: true reads the address the internet sees without rotating anything.

When the helper does not answer you get 409 device_not_provisioned_for_action.

Read that as "the phone did not answer in time." It is a timeout, despite the name: the box waits about thirty seconds for the phone to finish, and reports this when it does not. It does not mean:

• ❌ your key is missing a scope (no scope unlocks it; devices.control already covers all of it)
• ❌ this phone needs enabling for this one verb (there is no per-verb provisioning)
• ❌ there is a checkbox on the key-creation screen you missed (there is not)

Tell the two real cases apart by trying once more, about a minute later:

• It succeeds → the helper was busy, often because something else was driving that phone. Nothing is wrong.
• It refuses again → stop. Nothing on your side shortens that timeout, so further retries return the same answer. Report the phone's inventory_id, the verb, and the X-Request-Id. Pointer verbs still work on that phone, so use them if there is another route to the goal.

⚠️ open_url has a first-run cost, per phone and per destination. iOS asks whoever is holding the phone to approve each destination the first time that handset is asked to open it (Allow "ALI Remote" to open "Instagram"?). Until somebody taps it the call just waits, and at thirty seconds it is refused.

Measured on a live handset: first call to a new destination, 30 s and refused; once approved, about 2 s and it stays that way. A phone that opens Instagram in two seconds will still spend thirty being refused the first time you ask it for TikTok.

So the first open_url to a new (phone, destination) pair is slower than every later one. Boxes on current builds answer the prompt themselves, so it usually just costs a few extra seconds; where a box has not picked that up yet the first call is refused at thirty seconds instead. If a destination keeps being refused, ask ALI to approve that link on that phone rather than discovering it one refusal at a time in a live run.

Preflight: ask before you commit

Because the helper verbs share one dependency, one of them failing predicts the rest. So if a run is about to do something irreversible on a live account, probe first with the one that reads and changes nothing:

clipboard_get(device)   # succeeds  -> the helper answers; text and the switches will too
                        # 409       -> it is not answering in time; skip this phone

Note the preflight tells you about the helper, not about one verb: open_url can still be refused on a phone whose clipboard_get answers instantly, because it carries its own per-destination approval (see above).

This is the closest thing to a capability check, and it costs one cheap call against a phone rather than a half-finished post against an account.

The failure playbook

Branch on the stable code field, never on detail, which is prose and gets reworded. Every response carries X-Request-Id, successes included. Quote it in support tickets.

Driving well

Typing. Short plain ASCII goes over the keyboard. Anything longer, accented, or with emoji is routed through the phone's clipboard, which is a helper verb and therefore slower and fallible. Tap the field first: text goes wherever focus already is. text appends, so to replace existing content use press backspace with a repeat count first.

Gestures have one walk, drawn a little differently each time. Swipes, drags and scrolls are all the same sixteen-step walk, a little over half a second. Pacing parameters (duration_ms, steps, step_ms, brake, span) are accepted and ignored; they are deprecated and will be refused. Do not tune them, and do not conclude a gesture failed because it "went too fast".

Some drags need a long press first. hold_drag is the same walk after the press is held still, for the drags the phone only starts once it has decided the finger is staying put: rearranging home screen icons, or picking an item up to drop it somewhere else. It is never varied, since the press has to land on what it lifts. A slider or an app switcher card moves with a plain swipe, about a second sooner.

A swipe we lay out ourselves is varied. A swipe_direction with no distance is drawn the way a hand draws it: the line sits 5 to 15 % off to one side, each end moves 5 to 15 % of the travel, and the walk runs 5 to 15 % faster or slower. Everything where you gave the numbers is drawn as you gave it: swipe, drag, scroll, and a swipe_direction carrying a distance. precise: true turns the variation off and precise: false turns it on, on any of them. Two consequences worth planning for: a varied swipe never lands on the same pixel twice, so do not diff coordinates to check one happened, and precise: false needs the phone to have reported its screen size, so it is refused with 503 device_offline while it has not.

Directions are opposite on purpose. swipe_direction names where the finger travels: up drags from the bottom toward the top, which scrolls content down. scroll names where the view goes. Getting this backwards is the single most common gesture bug.

Force-closing an app. app_close closes whatever app is in front: it opens the App Switcher and flicks that app's card away. Open the app you mean first, since it does not choose which card to throw. It is pointer-driven, so it works on any casting phone even when the on-phone helper is not answering - unlike open_url and the other helper verbs.

Building it by hand is where people come unstuck: with more than one app open the front app's card is at the right of the switcher and the middle holds the app before it, so press appswitcher followed by a drag up the centre line force-closes a bystander. app_close reads the screen back before answering, so a success means the app that was in front is no longer showing - firmly on a still screen, less so on an app that animates anyway, where a whole-frame check cannot tell movement from a close; it answers effect_observed: false all the same, because "no longer in front" is what can be seen and "terminated" is not. A 422 command_rejected from it is not a retry: the first flick may already have thrown a different app's card, and sending it again throws another. Screenshot and look instead.

Taps landing off-target. Some handsets occasionally place a tap away from where it was aimed. The fix is calibrate, which re-centres the pointer. Send it when you see it happening, or once at the top of a long tapping sequence. Not before every tap - it is a round trip and it costs credits, so a tap-calibrate pair doubles both for a fault most phones do not have.

recast is expensive. On a phone that is already casting it bounces the session and the screen goes away while it reconnects. It answers as soon as the box accepts, not when the phone is back: poll casting. Budget up to two minutes; 15 to 30 seconds is the good case.

Credits come in three layers, and one of them is money. Every call is priced by what it does and spends, in order, from an included allowance that is free and comes back at UTC midnight, then from the agency's purchased balance, which is held until it is spent and pays only for the part of a day above the allowance. Over both sits a ceiling that refuses whether or not there is a balance, so a loop that goes wrong costs one day rather than the balance. The pool is per key, not per phone: the phone count only sizes it. Read your real numbers from GET /v1/usage rather than hardcoding any figure. A phone is physical hardware moving at its own pace; exceeding the limits does not make it faster.

Send an Idempotency-Key on every action you would not want to happen twice. A retry with the same key replays the first response instead of repeating the work.

Getting photos and videos onto phones

Files reach a phone through the agency's media vault, the same library the dashboard shows. The vault is the source of truth: put a file in once, then send it to as many phones as you like.

What to know before you rely on it:

• A push answers before the phone has the file. Delivery takes minutes. A push says which phones were queued and which were skipped, each with a reason; a phone somebody is using is skipped rather than interrupted. Follow up with get_media instead of assuming.
• The first file to a phone may wait on the handset. iOS can ask whoever holds the phone to allow it once. Until somebody does, that phone's deliveries fail, and so can the first removal. Report the phone; do not resend in a loop.
• Keep filenames short. The phone truncates long names, and a truncated one can land yet be recorded as lost.
• Delete from the phones before the vault. Deleting a vault file leaves every copy already sent where it is, and after that nothing can take it off through ALI.
• What was sent is our record, not the camera roll. list_device_media lists what we put on a phone. If a delivery says failed but the file may be there anyway, verify_media_on_devices re-reads the phone and corrects the record; do not push it again blind, or the phone collects duplicates.
• Files expire. The vault keeps a file for a few days and then lets it go; read its expiry rather than assuming it is still there next week.

Working safely on live accounts

These are the habits that separate an integration people trust from one that gets switched off.

• One phone first. Prove a flow end to end on a single handset before fanning out.
• Screenshot before and after anything irreversible - posting, sending, deleting, following. Keep both; they are your evidence that the right account did the right thing.
• Confirm the account before you act. The phone is a slot; the account signed into it is a fact about the world that can change without telling you. If your flow depends on being on a particular profile, look at the screen and verify it rather than trusting your mapping.
• Stop on the first unexpected screen. A login wall, a captcha, a "confirm your identity" prompt, an unexpected feed - these mean your assumptions are stale. Stopping costs one run. Carrying on costs an account.
• Never carry on from an unverified state. If you cannot tell what is on screen, stop and say so. "I could not confirm the post went out" is a useful answer. Guessing is not.
• Serialise per phone. One handset has one pointer. Two flows driving it at once produce interleaved taps that look like a possessed device.

Building an integration on top of this

Answers to the questions integrators actually ask, based on how the platform models things.

Account-to-device mapping. Keep it in your own store, keyed on the phone's public_id, not on its name and not on its position in a list. inventory_id (the physical sticker) is the stable human-facing identity and is what a person at the rack can read; public_id is the stable machine one. Names change, racks get re-ordered, and a phone can move between agencies. Re-resolve the mapping from GET /v1/devices at the start of every run rather than caching it across days.

Onboarding a new phone. A phone appears in the device list on its own once it is racked and casting. Before you trust it with an account: check casting, take a screenshot, run the clipboard_get preflight, and send one calibrate. If any of those fail, the handset is not ready and no amount of API-side work will make it ready.

Per-device queues. Serialise per phone, and make each job idempotent with an Idempotency-Key derived from the work, not from the attempt (so a retry reuses it). Do not build a global queue that lets two jobs reach one handset. Do not retry a 409 device_not_provisioned_for_action more than once - it is not a transient.

Execution proof. The API gives you three layers, weakest to strongest: the acknowledgement (device_ack), the X-Request-Id you can quote back to us, and a screenshot you took yourself. Only the third proves anything about the screen. For "was the right person on this phone at the time", GET /v1/analytics/sessions gives one row per person per phone per stretch, with the times, and GET /v1/analytics/members?include=devices gives the totals.

Connectivity. Phones drop and come back. Treat online/casting as facts that expire, check them at the start of a run rather than at the start of the day, and prefer skipping a phone over waking it aggressively.

When you are stuck

Say what you observed, not what you concluded. The useful report is:

Phone ph_… / sticker 1277. casting: true, snapshot fresh at 458x812. press home and press appswitcher acknowledged, new frames each time. open_url with instagram:// returned 409 device_not_provisioned_for_action, request id ea30dd66-…. Retried once after a minute, same result. Stopped - no further attempts, no posts, no account changes.

That report is exactly right, and with the first-run cost above in mind it points at the open_url approval for that destination before it points at a dead helper: the pointer verbs still answered, and it is instagram:// that was refused twice. A clipboard_get settles it, because that carries no per-destination approval: if it answers, the helper is fine and that link needs approving on that phone; if it refuses too, the helper itself is down. What the report should not ask for is a scope or a per-verb provisioning step, because neither exists.