I built a lead monitoring pipeline 5 times before it actually works

Most blog posts about automation start with the finished product. I'm going to start with every way this broke instead — because the failures are where the actual learning happened.

---

The problem I was trying to solve

Comfort Shooting, the marketing agency where I run the IT side of things, gets leads through Wix forms on the website. Before automation, someone had to manually check the email inbox for form submissions. Leads sat there for hours. Sometimes they disappeared completely.

The goal was simple: every time someone fills in a form on the site, the team gets an immediate notification in Discord. No delays. No manual checking. No lost leads.

Here's what it actually took to get there.

---

Attempt 1: The Wix Forms API (dead end)

The obvious approach — use the Wix Forms API to read form submissions directly.

I wrote a Python script, set up the API call, and immediately hit a wall: the Wix Forms API requires OAuth app-level tokens. The free tier doesn't grant access. Wix locks this behind their paid API program.

This is a genuine platform limitation — not a bug, not a configuration issue. Wix simply doesn't let free-tier sites read form data via their API.

Lesson: Always check API access requirements before writing code. Fifteen minutes of reading documentation saves hours of debugging.

---

Attempt 2: Scan the notification emails via IMAP (worked... mostly)

Since the form data lives in Wix's system and I can't access it directly, I went around it: read the notification emails that Wix sends to Gmail every time a form is submitted.

I built a Python IMAP scanner (check_wix_forms_email.py):

• Connects to Gmail via IMAP
• Searches the "Wix Noti" label for new messages
• Extracts lead data from each email
• Sends a formatted notification to Discord

It worked. For about a day.

Then it swallowed emails silently. No notifications. No errors. Just gone.

---

Attempt 3: v3, v4, v4.1 — the silent-swallowing bug

The problem was in the state file. The script tracks which emails it's already processed using a seen_ids.json file. Two things broke this:

1. The init-mode bug: When the script first runs, it marks all existing emails as "seen" to avoid spamming old leads. A logic error in v3 meant it would sometimes re-enter init mode and silently re-mark everything — including new emails that had just arrived.

2. State file deletion: During debugging, I accidentally deleted seen_ids.json. The script rebuilt it from the inbox, but the reconstruction logic had a subtle flaw — it missed emails that arrived while the file was missing.

I iterated through v3 → v4 → v4.1, adding:

• Meta-first guard (meta.json with initialized=true as the source of truth instead of seen_ids.json)
• Auto-recovery that rebuilds the seen set from IMAP if the file goes missing
• State integrity checks after every run
• Done backup after each execution

This version worked reliably for weeks. But polling every 10 minutes still meant a 0-10 minute delay on every lead.

---

Attempt 5: Real-time IMAP IDLE listener

A 10-minute delay on a lead notification is acceptable. A real-time system is better.

I rebuilt the whole pipeline around IMAP IDLE — a protocol where the server pushes new emails to the client instantly, instead of the client asking "anything new?" every few minutes.

What this replaced into:

• imap_idle_listener.py (518 lines) — persistent IMAP connection that listens for new emails and drops them into a delivery queue
• wix_delivery_queue.py — runs every minute, checks the queue, sends Discord notifications
• wix-idle-listener.service — systemd service with Restart=always, so the listener survives crashes and reboots
• Health check script running every 30 minutes to verify the listener is still alive
• Queue-before-seen atomicity — emails are written to the queue before being marked as processed, preventing the swallowing bug from v3

I also built in crash recovery: if the listener goes down and restarts, it processes any emails it missed during the outage.

---

What the system actually does today

Every time someone fills in a form on comfortshooting.com:

1. Wix sends a notification email to Gmail
2. The IMAP IDLE listener detects it within seconds
3. The email is queued for delivery
4. Within 1 minute, a formatted notification appears in Discord with the lead details
5. The health check confirms both systems are running

Total human involvement: zero. No one checks email. No one forwards leads. The agent handles it end to end.

---

What I actually learned

Fail fast, debug properly. Each version broke in a different way. The fixes weren't obvious — they required actually reading the failure, not guessing at it.

State management is the hardest part. The AI logic (read email, parse data, send notification) was easy. The hard part was making sure the system didn't lose track of what it had already processed.

Real-time requires different architecture. Polling and event-driven systems fail in different ways. Understanding which one you need before building saves a rebuild.

Wix's free tier is genuinely limited. This isn't a complaint — it's a business reality. Every platform has limitations. Work with them, not against them.

---

This post was written based on actual system logs and Obsidian vault records. No hypothetical scenarios. Everything described here is running on the live system today.