# Agenty Scraping Webhook

This script receives Agenty job-completion webhooks, maps the incoming Agenty agent to an internal dealer slug, writes the posted results to a CSV file, and updates the dealer JSON file list used by the feed pipeline.

## Current Flow

The active Agenty integration uses:

- `webhook.php`
- `config.php`

The older polling script `index.php` is now considered legacy. If retained, it should be treated as reference or fallback only, not as the primary Agenty feed process.

## How It Works

When Agenty sends a `POST` request to `webhook.php`, the script:

1. Checks the webhook token passed in the URL query string.
2. Reads `X-Agenty-AgentId` and `X-Agenty-JobId` from the request headers.
3. Maps the Agenty agent ID to a dealer slug using `config.php`.
4. Parses the JSON body and extracts the `result` rows.
5. Writes a CSV file to the dealer folder.
6. Updates `/var/www/feeds/json/{dealer_slug}_csv_file_list.json` with the latest filename.
7. Records the processed job so retries do not create duplicate CSV files.
8. Writes request activity to `logs/webhook.log`.

## Webhook URL

Use this webhook URL in Agenty:

```text
https://ftpfeeds.campers4sale.co.uk/agenty_scraping/agenty_scraping_script/webhook.php?token=95ba21005ae975ef6aa79c08aecb6d240ab772825ddb2a623dd3fd9a985ad3f9
```

## Dealer CSV Storage

CSV files are stored here:

```text
/var/www/feeds/agenty_scraping/uploads/agenty_scraping_csv/{dealer_slug}/{YYYY}/{MM}/{dealer_slug}_{YYYYMMDD}_{HHMM}.csv
```

Example:

```text
/var/www/feeds/agenty_scraping/uploads/agenty_scraping_csv/autocraftmotorcaravans/2026/03/autocraftmotorcaravans_20260315_1525.csv
```

## Dealer JSON File List

The latest CSV filename for each dealer is written here:

```text
/var/www/feeds/json/{dealer_slug}_csv_file_list.json
```

Example:

```text
/var/www/feeds/json/autocraftmotorcaravans_csv_file_list.json
```

Contents:

```json
["autocraftmotorcaravans_20260315_1525.csv"]
```

## Logs And Duplicate Protection

Files in `logs/`:

- `webhook.log`
  Request activity log for accepted, rejected, and duplicate webhook calls.
- `agenty_webhook_processed_jobs.json`
  State file used to remember processed `agent_id + job_id` combinations.

Old processed jobs are automatically pruned based on `processed_jobs_retention_days` in `config.php`.

## How To Add A New Dealer

1. Create or confirm the Agenty agent for that dealer.
2. Run the webhook once and inspect the request headers or Agenty UI to get the `X-Agenty-AgentId` value.
3. Open `config.php`.
4. Add the new mapping inside `agent_dealer_map`:

```php
'agent_dealer_map' => array(
    'avpuabzyff' => 'autocraftmotorcaravans',
    'newagentid123' => 'newdealer',
),
```

5. Save the file.
6. Trigger the Agenty webhook again.
7. Confirm:
   - CSV file is created in the dealer folder
   - JSON file list is updated
   - `logs/webhook.log` shows an accepted request

## Important Notes

- Agenty custom request headers are for scraping targets, not for webhook delivery.
- Dealer identification is handled internally by the `agent_dealer_map`.
- The webhook token is validated from the URL query string because Agenty does not support custom webhook headers for this use case.
- Duplicate webhook deliveries are ignored when the same `agent_id + job_id` is received again.

## Troubleshooting

If a webhook does not process:

1. Check `logs/webhook.log`.
2. Confirm the webhook URL includes the correct `token=...`.
3. Confirm the incoming `X-Agenty-AgentId` exists in `config.php`.
4. Confirm the JSON body contains `result` rows.
5. Check whether the job was skipped as a duplicate.