feat(n8n): resolve AI model credentials and project ID dynamically on deploy/test and update documentation
This commit is contained in:
parent
8bfd240860
commit
95b05c50f5
9 changed files with 593 additions and 46 deletions
|
|
@ -186,37 +186,40 @@ erDiagram
|
|||
|
||||
---
|
||||
|
||||
## 4. Workflows & n8n Integration Model
|
||||
### 4. Workflows & n8n Integration Model
|
||||
|
||||
By offloading calculations and integrations to n8n, we achieve a highly visual, modular, and editable workflow architecture.
|
||||
To achieve high modularity, testability, and adaptability, the system implements a **Thick Client (Next.js), Thin Coordinator (n8n)** architecture:
|
||||
* **Zero Database Connections in n8n**: n8n must not connect directly to PostgreSQL. All data read/write mutations are handled by Next.js API endpoints.
|
||||
* **Minimal Code Blocks in n8n**: Business logic, math calculations, and transaction management are kept on the Next.js side where they are type-safe and fully covered by unit tests.
|
||||
* **All AI/LLM inside n8n**: All LLM queries and AI-assisted audits are processed using native n8n LangChain and AI Agent nodes, allowing prompt/model hot-swapping without redeploying the Next.js codebase.
|
||||
* **Stalwart Email Notifications**: Email notifications are dispatched directly by n8n using native SMTP nodes configured to use the company's Stalwart mail server.
|
||||
|
||||
### 4.1. Sales Data Import Workflow
|
||||
1. **Trigger**: Next.js calls `POST /api/n8n/import-sales` which forwards the parsed Excel payload to n8n's webhook URL.
|
||||
1. **Trigger**: Next.js calls `POST /webhook/calculate-commissions` (or `/webhook-test/...` in dev) forwarding parsed Excel rows, uploader ID, and an idempotency key.
|
||||
2. **n8n Processing**:
|
||||
- Iterates through sales records.
|
||||
- Queries Next.js APIs to validate collaborator codes and hotel IDs.
|
||||
- Cleanses data and identifies duplicates.
|
||||
- Uses an LLM node (with configurable models: GPT-4, Claude 3.5, Gemini, etc.) to perform semantic anomaly checks (e.g., flag sales values that deviate more than 2.5 standard deviations from the collaborator's monthly average).
|
||||
3. **Response**: n8n POSTs the sanitized/flagged list back to `POST /api/sales/batch-save` in Next.js to update the database.
|
||||
- Queries `POST /api/n8n/validate-sales` to resolve database entities (collaborators, hotels) and retrieve historical standard deviation baselines.
|
||||
- Passes data to a native LangChain LLM Chain node (`@n8n/n8n-nodes-langchain.chainLlm`) connected to a **Primary Chat Model** (DeepSeek, `lmChatDeepSeek`) and a **Fallback Chat Model** (Google Gemini, `lmChatGoogleGemini`) to perform semantic anomaly validation and record anomaly flags. If the primary model fails or is rate-limited, n8n automatically falls back to the Google Gemini model.
|
||||
- **Dynamic Credential Mapping**: To support isolated project spaces, the bootstrap and test scripts query the n8n API (`GET /api/v1/credentials`) at deploy time, resolve the correct credential IDs for the DeepSeek and Gemini accounts in the workspace, and inject them into the workflow definition before creating or updating the workflow.
|
||||
3. **Response**: n8n calls `POST /api/sales/batch-save` (authenticated via `x-n8n-signature`) to persist finalized results in the PostgreSQL database under admin privileges.
|
||||
|
||||
### 4.2. Settlement Calculation Workflow
|
||||
1. **Trigger**: Next.js calls n8n to execute the calculation for period `YYYY-MM`.
|
||||
2. **n8n Processing**:
|
||||
- Pulls active plans, individual goals, and actual sales from the Next.js API.
|
||||
- Evaluates the mathematical formulas.
|
||||
- Evaluates rule scales and applies caps.
|
||||
- Generates notifications (via email or push notifications) using n8n integrations.
|
||||
3. **Response**: Updates database records via the Next.js API and completes the task.
|
||||
- Gathers plans, goals, and sales results from `/api/n8n/fetch-calculation-data?period=YYYY-MM`.
|
||||
- Offloads calculation processing to `/api/n8n/process-formula` to run the type-safe mathematical rules engine (tiers, bonuses, clawbacks).
|
||||
- Runs a native AI Agent node to audit calculated payouts for compliance (e.g. capping rules, negative payouts).
|
||||
- Dispatches email alerts using the Stalwart mail server if any extreme payouts require analyst reviews.
|
||||
3. **Response**: Persists settlements via `POST /api/n8n/save-settlements` with static signature checks.
|
||||
|
||||
### 4.3. Test Webhook Branching & Database Isolation in n8n
|
||||
To ensure complete isolation of production data, all n8n workflows must follow a strict testing branch architecture:
|
||||
### 4.3. Test Webhook Branching & Environment Isolation in n8n
|
||||
To ensure complete isolation of production data, all n8n workflows follow a strict testing branch architecture:
|
||||
1. **Webhook Entry Node**: n8n listens on two webhook path variants:
|
||||
- Production calls hit: `/webhook/calculate-commissions`
|
||||
- Test suite calls hit: `/webhook-test/calculate-commissions`
|
||||
- Production calls hit: `/webhook/calculate-commissions` or `/webhook/calculate-settlements`
|
||||
- Test suite calls hit: `/webhook-test/calculate-commissions` or `/webhook-test/calculate-settlements`
|
||||
2. **Conditional Path Routing**:
|
||||
- An `IF` node immediately checks if the webhook request path contains `webhook-test`.
|
||||
- **True (Test Mode)**: The workflow overrides its database credential node configurations to connect to `TEST_DATABASE_URL` (the secondary test sandbox database) and makes API callbacks back to the Next.js test instance.
|
||||
- **False (Prod Mode)**: The workflow executes against the main `DATABASE_URL` and interacts with the production Next.js instance.
|
||||
- **True (Test Mode)**: The workflow sets its Next.js target host variables to the development instance (`https://special-hotel-dev.gaboggamer.online`), which uses `TEST_DATABASE_URL` (the secondary test sandbox database) and signing secret `N8N_WEBHOOK_SECRET_DEV`.
|
||||
- **False (Prod Mode)**: The workflow executes against the production instance, validating calls using `N8N_WEBHOOK_SECRET`.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -57,10 +57,14 @@ graph TD
|
|||
```
|
||||
|
||||
### 1.4. n8n Integration & Callback Validation
|
||||
For automatic integrations (US-COM-005):
|
||||
1. **Webhook Security**: Next.js exposes a public-facing but secured callback endpoint `/api/sales/batch-save`.
|
||||
2. **Signature Verification**: Webhooks from n8n must include the `x-n8n-signature` header. The server verifies this token against `N8N_WEBHOOK_SECRET` before executing database updates.
|
||||
3. **Async Status Polling**: The frontend displays a premium progress UI. If routed through n8n, it polls `GET /api/sales/import/status/[key]` to check database status and update progress.
|
||||
For automatic integrations (US-COM-005), the workflow follows a **Thick Client (Next.js), Thin Coordinator (n8n)** architecture:
|
||||
1. **Zero DB Connections & Zero Code Blocks**: n8n does not connect directly to the database or run heavy processing code. Instead, n8n invokes Next.js validation and logic endpoints, using n8n strictly for pipeline orchestration and third-party notifications.
|
||||
2. **Encapsulated LLM/AI & Model Fallback**: All LLM-based checks (semantic anomaly checks) are executed directly inside n8n using native nodes. The workflow uses a **Primary Chat Model** node (DeepSeek) and a **Fallback Chat Model** node (Google Gemini) hooked to a LangChain LLM Chain node with failover enabled.
|
||||
3. **Dynamic Workspace Credential Resolution**: During the bootstrap and E2E test runs, the system queries `GET /api/v1/credentials` from n8n, searches for active credentials matching the DeepSeek and Gemini types (`deepSeekApi` and `googlePalmApi`), and dynamically injects their IDs and names into the workflow JSON before deployment.
|
||||
4. **Webhook Security**: Next.js exposes a public-facing but secured callback endpoint `/api/sales/batch-save`.
|
||||
5. **Signature Verification**: Webhooks from n8n must include the `x-n8n-signature` header. The server verifies this token against `N8N_WEBHOOK_SECRET` before executing database updates.
|
||||
6. **Stalwart SMTP Server**: All outbound notifications (such as system error reports or settlement anomalies alerts) are sent by n8n using native SMTP/Email nodes pointing to the Stalwart mail server.
|
||||
7. **Async Status Polling**: The frontend displays a progress UI. If routed through n8n, it polls `GET /api/sales/import/status/[key]` to check database status and update progress.
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -90,23 +90,10 @@
|
|||
400
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsCode": "// Process individual sales records and run standard deviation / anomaly checks\n// Output format matches batch-save API schema\nconst webhookNode = $node[\"Webhook\"];\nconst body = webhookNode ? webhookNode.json.body : {};\nconst sales = body.sales || [];\nconst idempotencyKey = body.idempotencyKey;\nconst uploaderId = body.uploaderId;\nconst appUrl = items[0].json.appUrl;\nconst signature = items[0].json.signature;\n\n// We will simulate anomaly detection:\n// Sales records deviating abnormally from standard amounts are marked or adjusted\nconst processedSales = sales.map(sale => {\n const isAnomaly = sale.amount > 1000000; // Example threshold (>1M)\n return {\n ...sale,\n isAnomaly,\n flaggedReason: isAnomaly ? 'Sales amount exceeds normal threshold limits (>1M)' : null\n };\n});\n\nreturn [{\n json: {\n idempotencyKey,\n uploaderId,\n sales: processedSales,\n appUrl,\n signature\n }\n}];"
|
||||
},
|
||||
"id": "Code-Node-1",
|
||||
"name": "Data Validation & Anomaly Checks",
|
||||
"type": "n8n-nodes-base.code",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
750,
|
||||
300
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "={{$node[\"Data Validation & Anomaly Checks\"].json[\"appUrl\"]}}/api/sales/batch-save",
|
||||
"url": "={{ $('Set Dev Environment').isExecuted ? $('Set Dev Environment').item.json.appUrl : $('Set Prod Environment').item.json.appUrl }}/api/n8n/validate-sales",
|
||||
"sendHeaders": true,
|
||||
"headerParameters": {
|
||||
"parameters": [
|
||||
|
|
@ -116,28 +103,127 @@
|
|||
},
|
||||
{
|
||||
"name": "x-n8n-signature",
|
||||
"value": "={{$node[\"Data Validation & Anomaly Checks\"].json[\"signature\"]}}"
|
||||
"value": "={{ $('Set Dev Environment').isExecuted ? $('Set Dev Environment').item.json.signature : $('Set Prod Environment').item.json.signature }}"
|
||||
}
|
||||
]
|
||||
},
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={{ { idempotencyKey: $node[\"Data Validation & Anomaly Checks\"].json[\"idempotencyKey\"], uploaderId: $node[\"Data Validation & Anomaly Checks\"].json[\"uploaderId\"], sales: $node[\"Data Validation & Anomaly Checks\"].json[\"sales\"] } }}",
|
||||
"jsonBody": "={\n \"sales\": {{ JSON.stringify($node[\"Webhook\"].json[\"body\"][\"sales\"]) }}\n}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "Http-Request-Validate",
|
||||
"name": "HTTP Request: validate-sales",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4.1,
|
||||
"position": [
|
||||
700,
|
||||
300
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"promptType": "define",
|
||||
"text": "=Verify the following sales records for anomalies. Return a JSON object containing a \"sales\" array, matching the input structure, but adding the fields \"isAnomaly\" (boolean) and \"flaggedReason\" (string or null) for each record.\nIf a record has isValid: false, set isAnomaly to true and flaggedReason to invalidReason.\nIf the amount is greater than the threshold, set isAnomaly to true and flaggedReason to \"Sales amount deviates abnormally from collaborator monthly average\".\nIf the amount is extremely high (e.g. >1000000), set isAnomaly to true and flaggedReason to \"Sales amount exceeds normal threshold limits (>1M)\".\nOtherwise, set isAnomaly to false and flaggedReason to null.\n\nInput sales data:\n{{ JSON.stringify($node[\"HTTP Request: validate-sales\"].json.validationResults) }}",
|
||||
"hasOutputParser": true,
|
||||
"needsFallback": true,
|
||||
"options": {}
|
||||
},
|
||||
"id": "LLM-Chain-Anomaly",
|
||||
"name": "LLM Chain: Anomaly Detection",
|
||||
"type": "@n8n/n8n-nodes-langchain.chainLlm",
|
||||
"typeVersion": 1.9,
|
||||
"position": [
|
||||
900,
|
||||
300
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"options": {}
|
||||
},
|
||||
"id": "DeepSeek-Model-Node",
|
||||
"name": "Primary Chat Model",
|
||||
"type": "@n8n/n8n-nodes-langchain.lmChatDeepSeek",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
750,
|
||||
480
|
||||
],
|
||||
"credentials": {
|
||||
"deepSeekApi": {
|
||||
"id": "YbGKBhQ9rlUebE4F",
|
||||
"name": "Semillero2_Primary_deepSeekApi"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"options": {}
|
||||
},
|
||||
"id": "Google-Gemini-Model-Node",
|
||||
"name": "Fallback Chat Model",
|
||||
"type": "@n8n/n8n-nodes-langchain.lmChatGoogleGemini",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
900,
|
||||
480
|
||||
],
|
||||
"credentials": {
|
||||
"googlePalmApi": {
|
||||
"id": "1Vz05qteUH9kLf7Z",
|
||||
"name": "Google Gemini(PaLM) Api account"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"jsonSchema": "{\n \"type\": \"object\",\n \"properties\": {\n \"sales\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"object\",\n \"properties\": {\n \"username\": { \"type\": \"string\" },\n \"hotelCode\": { \"type\": \"string\" },\n \"period\": { \"type\": \"string\" },\n \"amount\": { \"type\": \"number\" },\n \"salesCount\": { \"type\": \"number\" },\n \"transactionId\": { \"type\": \"string\" },\n \"isAnomaly\": { \"type\": \"boolean\" },\n \"flaggedReason\": { \"type\": \"string\" }\n },\n \"required\": [\"username\", \"hotelCode\", \"period\", \"amount\", \"salesCount\", \"isAnomaly\"]\n }\n }\n },\n \"required\": [\"sales\"]\n}"
|
||||
},
|
||||
"id": "Structured-Output-Parser-Node",
|
||||
"name": "Structured Output Parser",
|
||||
"type": "@n8n/n8n-nodes-langchain.outputParserStructured",
|
||||
"typeVersion": 1,
|
||||
"position": [
|
||||
1050,
|
||||
480
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"method": "POST",
|
||||
"url": "={{ $('Set Dev Environment').isExecuted ? $('Set Dev Environment').item.json.appUrl : $('Set Prod Environment').item.json.appUrl }}/api/sales/batch-save",
|
||||
"sendHeaders": true,
|
||||
"headerParameters": {
|
||||
"parameters": [
|
||||
{
|
||||
"name": "Content-Type",
|
||||
"value": "application/json"
|
||||
},
|
||||
{
|
||||
"name": "x-n8n-signature",
|
||||
"value": "={{ $('Set Dev Environment').isExecuted ? $('Set Dev Environment').item.json.signature : $('Set Prod Environment').item.json.signature }}"
|
||||
}
|
||||
]
|
||||
},
|
||||
"sendBody": true,
|
||||
"specifyBody": "json",
|
||||
"jsonBody": "={\n \"idempotencyKey\": \"{{ $node[\"Webhook\"].json[\"body\"][\"idempotencyKey\"] }}\",\n \"uploaderId\": {{ $node[\"Webhook\"].json[\"body\"][\"uploaderId\"] }},\n \"sales\": {{ JSON.stringify($node[\"LLM Chain: Anomaly Detection\"].json[\"sales\"]) }}\n}",
|
||||
"options": {}
|
||||
},
|
||||
"id": "Http-Request-1",
|
||||
"name": "Callback batch-save",
|
||||
"type": "n8n-nodes-base.httpRequest",
|
||||
"typeVersion": 4,
|
||||
"typeVersion": 4.1,
|
||||
"position": [
|
||||
950,
|
||||
1200,
|
||||
300
|
||||
]
|
||||
},
|
||||
{
|
||||
"parameters": {
|
||||
"respondWith": "json",
|
||||
"responseBody": "={\n \"success\": true,\n \"code\": \"IMPORT_ACCEPTED\",\n \"metadata\": {\n \"idempotencyKey\": \"{{$node[\"Webhook\"].json[\"body\"][\"idempotencyKey\"]}}\",\n \"status\": \"PROCESSING\"\n }\n}"
|
||||
"responseBody": "={\n \"success\": true,\n \"code\": \"IMPORT_ACCEPTED\",\n \"metadata\": {\n \"idempotencyKey\": \"{{ $node[\"Webhook\"].json[\"body\"][\"idempotencyKey\"] }}\",\n \"status\": \"PROCESSING\"\n }\n}"
|
||||
},
|
||||
"id": "Respond-To-Webhook",
|
||||
"name": "Respond to Webhook",
|
||||
|
|
@ -188,7 +274,7 @@
|
|||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Data Validation & Anomaly Checks",
|
||||
"node": "HTTP Request: validate-sales",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
|
|
@ -199,14 +285,58 @@
|
|||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "Data Validation & Anomaly Checks",
|
||||
"node": "HTTP Request: validate-sales",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Data Validation & Anomaly Checks": {
|
||||
"HTTP Request: validate-sales": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
"node": "LLM Chain: Anomaly Detection",
|
||||
"type": "main",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Primary Chat Model": {
|
||||
"ai_languageModel": [
|
||||
[
|
||||
{
|
||||
"node": "LLM Chain: Anomaly Detection",
|
||||
"type": "ai_languageModel",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Fallback Chat Model": {
|
||||
"ai_languageModel": [
|
||||
[
|
||||
{
|
||||
"node": "LLM Chain: Anomaly Detection",
|
||||
"type": "ai_languageModel",
|
||||
"index": 1
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"Structured Output Parser": {
|
||||
"outputParser": [
|
||||
[
|
||||
{
|
||||
"node": "LLM Chain: Anomaly Detection",
|
||||
"type": "ai_outputParser",
|
||||
"index": 0
|
||||
}
|
||||
]
|
||||
]
|
||||
},
|
||||
"LLM Chain: Anomaly Detection": {
|
||||
"main": [
|
||||
[
|
||||
{
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@
|
|||
"lint": "eslint",
|
||||
"db:seed": "npx prisma db execute --file prisma/rls_and_seed.sql",
|
||||
"db:seed-test": "DATABASE_URL=\"postgresql://special_hotel_user:SpecialHotel1235%2F%2A-%2B@pg.gaboggamer.online/special_hotel_test?schema=public\" npx prisma db execute --file prisma/rls_and_seed.sql",
|
||||
"n8n:bootstrap": "node scripts/n8n-bootstrap.js",
|
||||
"test:rls": "node prisma/test-rls.js",
|
||||
"test:auth-rls": "node prisma/test-auth-rls.js",
|
||||
"test:ui": "next build && node prisma/test-phase3-ui.js && node prisma/test-phase4-ui.js",
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 54 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 54 KiB After Width: | Height: | Size: 54 KiB |
|
|
@ -137,6 +137,88 @@ async function runTests() {
|
|||
}
|
||||
}
|
||||
|
||||
// 3.5. Fetch existing workflows to retrieve the project ID dynamically
|
||||
console.log("Fetching project ID from existing workflows...");
|
||||
let projectId;
|
||||
try {
|
||||
const listRes = await fetch(`${n8nUrl}/api/v1/workflows`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
}
|
||||
});
|
||||
if (listRes.ok) {
|
||||
const listData = await listRes.json();
|
||||
const firstWf = listData.data.find(w => w.shared && w.shared.length > 0);
|
||||
if (firstWf) {
|
||||
projectId = firstWf.shared[0].projectId;
|
||||
console.log(`Resolved project ID dynamically: ${projectId}`);
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn("Warning: Could not fetch project ID dynamically, falling back to default.", err.message);
|
||||
}
|
||||
|
||||
// 3.6. Fetch credentials dynamically to resolve AI accounts
|
||||
console.log("Fetching credentials from n8n to resolve AI accounts...");
|
||||
let deepSeekCred = null;
|
||||
let geminiCred = null;
|
||||
try {
|
||||
const credsRes = await fetch(`${n8nUrl}/api/v1/credentials`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
}
|
||||
});
|
||||
if (credsRes.ok) {
|
||||
const credsData = await credsRes.json();
|
||||
const credentials = Array.isArray(credsData.data) ? credsData.data : (credsData.data.credentials || []);
|
||||
|
||||
const deepSeekCreds = credentials.filter(c => c.type === 'deepSeekApi');
|
||||
if (deepSeekCreds.length > 0) {
|
||||
deepSeekCred = deepSeekCreds.find(c => c.name.toLowerCase().includes('deepseek')) || deepSeekCreds[0];
|
||||
}
|
||||
|
||||
const geminiCreds = credentials.filter(c => c.type === 'googlePalmApi');
|
||||
if (geminiCreds.length > 0) {
|
||||
geminiCred = geminiCreds.find(c => c.name.toLowerCase().includes('gemini') || c.name.toLowerCase().includes('google')) || geminiCreds[0];
|
||||
}
|
||||
|
||||
if (deepSeekCred) {
|
||||
console.log(`Resolved DeepSeek credential: "${deepSeekCred.name}" (ID: ${deepSeekCred.id})`);
|
||||
}
|
||||
if (geminiCred) {
|
||||
console.log(`Resolved Gemini credential: "${geminiCred.name}" (ID: ${geminiCred.id})`);
|
||||
}
|
||||
} else {
|
||||
console.warn(`Warning: Failed to fetch credentials. Status: ${credsRes.status}`);
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn("Warning: Could not fetch credentials dynamically:", err.message);
|
||||
}
|
||||
|
||||
// Inject resolved credentials dynamically into the workflow JSON
|
||||
for (const node of workflowJson.nodes) {
|
||||
if (node.type === '@n8n/n8n-nodes-langchain.lmChatDeepSeek' && deepSeekCred) {
|
||||
node.credentials = {
|
||||
deepSeekApi: {
|
||||
id: deepSeekCred.id,
|
||||
name: deepSeekCred.name
|
||||
}
|
||||
};
|
||||
console.log(`Injected DeepSeek credential into node "${node.name}"`);
|
||||
}
|
||||
if (node.type === '@n8n/n8n-nodes-langchain.lmChatGoogleGemini' && geminiCred) {
|
||||
node.credentials = {
|
||||
googlePalmApi: {
|
||||
id: geminiCred.id,
|
||||
name: geminiCred.name
|
||||
}
|
||||
};
|
||||
console.log(`Injected Gemini credential into node "${node.name}"`);
|
||||
}
|
||||
}
|
||||
|
||||
// 4. Create workflow in n8n
|
||||
console.log("Deploying workflow to real n8n instance...");
|
||||
const workflowName = `Semillero E2E Integration: Sales Import - ${Date.now()}`;
|
||||
|
|
@ -150,7 +232,8 @@ async function runTests() {
|
|||
name: workflowName,
|
||||
nodes: workflowJson.nodes,
|
||||
connections: workflowJson.connections,
|
||||
settings: workflowJson.settings || {}
|
||||
settings: workflowJson.settings || {},
|
||||
projectId
|
||||
})
|
||||
});
|
||||
|
||||
|
|
|
|||
195
scripts/n8n-bootstrap.js
Normal file
195
scripts/n8n-bootstrap.js
Normal file
|
|
@ -0,0 +1,195 @@
|
|||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
require('dotenv').config();
|
||||
|
||||
async function bootstrap() {
|
||||
console.log("=== STARTING N8N WORKFLOW BOOTSTRAP / KICKSTART ===");
|
||||
|
||||
// 1. Fetch n8n API configuration from .agents/mcp_config.json
|
||||
const mcpConfigPath = path.join(__dirname, '../.agents/mcp_config.json');
|
||||
if (!fs.existsSync(mcpConfigPath)) {
|
||||
console.error("Error: .agents/mcp_config.json not found.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const mcpConfig = JSON.parse(fs.readFileSync(mcpConfigPath, 'utf8'));
|
||||
const n8nEnv = mcpConfig.mcpServers.n8n.env;
|
||||
const n8nUrl = n8nEnv.N8N_API_URL || "https://n8n.gaboggamer.online";
|
||||
const n8nApiKey = n8nEnv.N8N_API_KEY;
|
||||
|
||||
if (!n8nApiKey) {
|
||||
console.error("Error: N8N_API_KEY is missing in mcp_config.json.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
// 2. Load the official local workflow JSON
|
||||
const workflowPath = path.join(__dirname, '../n8n/sales_import_workflow.json');
|
||||
if (!fs.existsSync(workflowPath)) {
|
||||
console.error("Error: n8n/sales_import_workflow.json not found.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const workflowJson = JSON.parse(fs.readFileSync(workflowPath, 'utf8'));
|
||||
const targetName = workflowJson.name || "Sales Data Import & Validation";
|
||||
|
||||
// 2.5. Fetch credentials dynamically to resolve AI accounts
|
||||
console.log("Fetching credentials from n8n to resolve AI accounts...");
|
||||
let deepSeekCred = null;
|
||||
let geminiCred = null;
|
||||
try {
|
||||
const credsRes = await fetch(`${n8nUrl}/api/v1/credentials`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
}
|
||||
});
|
||||
if (credsRes.ok) {
|
||||
const credsData = await credsRes.json();
|
||||
const credentials = Array.isArray(credsData.data) ? credsData.data : (credsData.data.credentials || []);
|
||||
|
||||
const deepSeekCreds = credentials.filter(c => c.type === 'deepSeekApi');
|
||||
if (deepSeekCreds.length > 0) {
|
||||
deepSeekCred = deepSeekCreds.find(c => c.name.toLowerCase().includes('deepseek')) || deepSeekCreds[0];
|
||||
}
|
||||
|
||||
const geminiCreds = credentials.filter(c => c.type === 'googlePalmApi');
|
||||
if (geminiCreds.length > 0) {
|
||||
geminiCred = geminiCreds.find(c => c.name.toLowerCase().includes('gemini') || c.name.toLowerCase().includes('google')) || geminiCreds[0];
|
||||
}
|
||||
|
||||
if (deepSeekCred) {
|
||||
console.log(`Resolved DeepSeek credential: "${deepSeekCred.name}" (ID: ${deepSeekCred.id})`);
|
||||
}
|
||||
if (geminiCred) {
|
||||
console.log(`Resolved Gemini credential: "${geminiCred.name}" (ID: ${geminiCred.id})`);
|
||||
}
|
||||
} else {
|
||||
console.warn(`Warning: Failed to fetch credentials. Status: ${credsRes.status}`);
|
||||
}
|
||||
} catch (err) {
|
||||
console.warn("Warning: Could not fetch credentials dynamically:", err.message);
|
||||
}
|
||||
|
||||
// Inject resolved credentials dynamically into the workflow JSON
|
||||
for (const node of workflowJson.nodes) {
|
||||
if (node.type === '@n8n/n8n-nodes-langchain.lmChatDeepSeek' && deepSeekCred) {
|
||||
node.credentials = {
|
||||
deepSeekApi: {
|
||||
id: deepSeekCred.id,
|
||||
name: deepSeekCred.name
|
||||
}
|
||||
};
|
||||
console.log(`Injected DeepSeek credential into node "${node.name}"`);
|
||||
}
|
||||
if (node.type === '@n8n/n8n-nodes-langchain.lmChatGoogleGemini' && geminiCred) {
|
||||
node.credentials = {
|
||||
googlePalmApi: {
|
||||
id: geminiCred.id,
|
||||
name: geminiCred.name
|
||||
}
|
||||
};
|
||||
console.log(`Injected Gemini credential into node "${node.name}"`);
|
||||
}
|
||||
}
|
||||
|
||||
// 3. Check if workflow already exists in n8n
|
||||
console.log(`Searching for existing workflow named "${targetName}"...`);
|
||||
const listRes = await fetch(`${n8nUrl}/api/v1/workflows`, {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
}
|
||||
});
|
||||
|
||||
if (!listRes.ok) {
|
||||
console.error(`Failed to list workflows from n8n. Status: ${listRes.status}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const listData = await listRes.json();
|
||||
const existingWorkflow = listData.data.find(w => w.name === targetName);
|
||||
|
||||
// Retrieve project ID from existing workflows if available
|
||||
let projectId;
|
||||
const firstWf = listData.data.find(w => w.shared && w.shared.length > 0);
|
||||
if (firstWf) {
|
||||
projectId = firstWf.shared[0].projectId;
|
||||
console.log(`Resolved project ID dynamically: ${projectId}`);
|
||||
}
|
||||
|
||||
let workflowId;
|
||||
if (existingWorkflow) {
|
||||
workflowId = existingWorkflow.id;
|
||||
console.log(`Found existing workflow. ID: ${workflowId}. Updating...`);
|
||||
|
||||
const updateRes = await fetch(`${n8nUrl}/api/v1/workflows/${workflowId}`, {
|
||||
method: 'PUT',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
},
|
||||
body: JSON.stringify({
|
||||
name: targetName,
|
||||
nodes: workflowJson.nodes,
|
||||
connections: workflowJson.connections,
|
||||
settings: workflowJson.settings || {}
|
||||
})
|
||||
});
|
||||
|
||||
if (!updateRes.ok) {
|
||||
console.error(`Failed to update workflow. Status: ${updateRes.status}`);
|
||||
process.exit(1);
|
||||
}
|
||||
console.log(`Workflow updated successfully!`);
|
||||
} else {
|
||||
console.log("No existing workflow found. Creating new workflow...");
|
||||
|
||||
const createRes = await fetch(`${n8nUrl}/api/v1/workflows`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
},
|
||||
body: JSON.stringify({
|
||||
name: targetName,
|
||||
nodes: workflowJson.nodes,
|
||||
connections: workflowJson.connections,
|
||||
settings: workflowJson.settings || {},
|
||||
projectId
|
||||
})
|
||||
});
|
||||
|
||||
if (!createRes.ok) {
|
||||
console.error(`Failed to create workflow. Status: ${createRes.status}`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
const createData = await createRes.json();
|
||||
workflowId = createData.id;
|
||||
console.log(`Workflow created successfully! ID: ${workflowId}`);
|
||||
}
|
||||
|
||||
// 4. Activate the workflow
|
||||
console.log(`Activating workflow ${workflowId}...`);
|
||||
const activateRes = await fetch(`${n8nUrl}/api/v1/workflows/${workflowId}/activate`, {
|
||||
method: 'POST',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-N8N-API-KEY': n8nApiKey
|
||||
},
|
||||
body: JSON.stringify({})
|
||||
});
|
||||
|
||||
if (!activateRes.ok) {
|
||||
console.error(`Warning: Failed to activate workflow. Status: ${activateRes.status}`);
|
||||
} else {
|
||||
console.log(`Workflow activated and ready!`);
|
||||
}
|
||||
|
||||
console.log("=== N8N WORKFLOW BOOTSTRAP COMPLETED SUCCESSFULLY ===");
|
||||
}
|
||||
|
||||
bootstrap().catch(err => {
|
||||
console.error("Fatal bootstrap error:", err);
|
||||
process.exit(1);
|
||||
});
|
||||
131
src/app/api/n8n/validate-sales/route.ts
Normal file
131
src/app/api/n8n/validate-sales/route.ts
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
import { NextRequest, NextResponse } from 'next/server';
|
||||
import { getPrisma } from '@/lib/db';
|
||||
import crypto from 'crypto';
|
||||
|
||||
function safeCompare(a: string, b: string): boolean {
|
||||
const bufA = Buffer.from(a);
|
||||
const bufB = Buffer.from(b);
|
||||
if (bufA.length !== bufB.length) {
|
||||
crypto.timingSafeEqual(bufA, bufA);
|
||||
return false;
|
||||
}
|
||||
return crypto.timingSafeEqual(bufA, bufB);
|
||||
}
|
||||
|
||||
export async function POST(req: NextRequest) {
|
||||
try {
|
||||
const signature = req.headers.get('x-n8n-signature') || '';
|
||||
const expectedSecret = process.env.N8N_WEBHOOK_SECRET || '';
|
||||
|
||||
if (!expectedSecret || !signature || !safeCompare(signature, expectedSecret)) {
|
||||
return NextResponse.json({
|
||||
success: false,
|
||||
error: {
|
||||
code: 'UNAUTHORIZED',
|
||||
metadata: { message: 'Invalid or missing signature.' }
|
||||
}
|
||||
}, { status: 401 });
|
||||
}
|
||||
|
||||
const body = await req.json();
|
||||
const { sales } = body;
|
||||
|
||||
if (!Array.isArray(sales)) {
|
||||
return NextResponse.json({
|
||||
success: false,
|
||||
error: {
|
||||
code: 'BAD_REQUEST',
|
||||
metadata: { message: 'sales must be an array.' }
|
||||
}
|
||||
}, { status: 400 });
|
||||
}
|
||||
|
||||
const prisma = getPrisma();
|
||||
|
||||
// Query all unique collaborator usernames and hotel codes to validate them
|
||||
const uniqueUsernames = Array.from(new Set(sales.map(s => s.username).filter(Boolean))) as string[];
|
||||
const uniqueHotelCodes = Array.from(new Set(sales.map(s => s.hotelCode).filter(Boolean))) as string[];
|
||||
|
||||
// Bypass RLS context
|
||||
const dbUsers = await prisma.user.findMany({
|
||||
where: { username: { in: uniqueUsernames } }
|
||||
});
|
||||
const dbHotels = await prisma.hotel.findMany({
|
||||
where: { code: { in: uniqueHotelCodes } }
|
||||
});
|
||||
|
||||
const userMap = new Map<string, any>(dbUsers.map(u => [u.username, u]));
|
||||
const hotelMap = new Map<string, any>(dbHotels.map(h => [h.code, h]));
|
||||
|
||||
const validationResults = [];
|
||||
|
||||
for (const s of sales) {
|
||||
const user = userMap.get(s.username);
|
||||
const hotel = hotelMap.get(s.hotelCode);
|
||||
|
||||
if (!user) {
|
||||
validationResults.push({
|
||||
...s,
|
||||
isValid: false,
|
||||
invalidReason: 'COLLABORATOR_NOT_FOUND',
|
||||
avgAmount: 0,
|
||||
stdDev: 0,
|
||||
threshold: 0,
|
||||
historicalCount: 0
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
if (!hotel) {
|
||||
validationResults.push({
|
||||
...s,
|
||||
isValid: false,
|
||||
invalidReason: 'HOTEL_NOT_FOUND',
|
||||
avgAmount: 0,
|
||||
stdDev: 0,
|
||||
threshold: 0,
|
||||
historicalCount: 0
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Fetch historical sales results for this user to compute standard deviation
|
||||
const historicalSales = await prisma.salesResult.findMany({
|
||||
where: { userId: user.id },
|
||||
select: { amount: true }
|
||||
});
|
||||
|
||||
const amounts = historicalSales.map(h => Number(h.amount));
|
||||
const count = amounts.length;
|
||||
|
||||
const avgAmount = count > 0 ? amounts.reduce((a, b) => a + b, 0) / count : Number(s.amount);
|
||||
const variance = count > 1 ? amounts.reduce((a, b) => a + Math.pow(b - avgAmount, 2), 0) / (count - 1) : 0;
|
||||
const stdDev = count > 1 ? Math.sqrt(variance) : 0;
|
||||
const threshold = avgAmount + 2.5 * stdDev;
|
||||
|
||||
validationResults.push({
|
||||
...s,
|
||||
isValid: true,
|
||||
avgAmount,
|
||||
stdDev,
|
||||
threshold,
|
||||
historicalCount: count
|
||||
});
|
||||
}
|
||||
|
||||
return NextResponse.json({
|
||||
success: true,
|
||||
validationResults
|
||||
}, { status: 200 });
|
||||
|
||||
} catch (err: any) {
|
||||
console.error('validate-sales error:', err);
|
||||
return NextResponse.json({
|
||||
success: false,
|
||||
error: {
|
||||
code: 'VALIDATE_SALES_FAILED',
|
||||
metadata: { message: err.message }
|
||||
}
|
||||
}, { status: 500 });
|
||||
}
|
||||
}
|
||||
Loading…
Reference in a new issue