Skip to content

Commit 1eda44d

Browse files
waleedlatif1waleedlatif1
authored
imrpovement(kb): added client-side store, shared utils, background processing, retry with exponential backoff to knowledge base (#453)
* added background processing for file upload to knowledge base, added retry with exponential backoff, individual file retry, statuses * added knowledge store * added knowledge base selector and docs for knowledge block * fixed build * fix contributors page * significantly improved error handling, typing, fault tolerance * standardize file icon size
1 parent 282345b commit 1eda44d

File tree

39 files changed

+6861
-1355
lines changed

39 files changed

+6861
-1355
lines changed

‎apps/docs/content/docs/tools/knowledge.mdx‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
title: Knowledge
3+
description: Search knowledge
4+
---
5+
6+
import { BlockInfoCard } from "@/components/ui/block-info-card"
7+
8+
<BlockInfoCard
9+
type="knowledge"
10+
color="#00B0B0"
11+
icon={true}
12+
iconSvg={`<svg className="block-icon"
13+
14+
xmlns='http://www.w3.org/2000/svg'
15+
16+
17+
viewBox='0 0 24 24'
18+
fill='none'
19+
stroke='currentColor'
20+
strokeWidth='1.5'
21+
strokeLinecap='round'
22+
strokeLinejoin='round'
23+
>
24+
<path d='M21 10V8a2 2 0 0 0-1-1.73l-7-4a2 2 0 0 0-2 0l-7 4A2 2 0 0 0 3 8v8a2 2 0 0 0 1 1.73l7 4a2 2 0 0 0 2 0l2-1.14' />
25+
<path d='m7.5 4.27 9 5.15' />
26+
<polyline points='3.29 7 12 12 20.71 7' />
27+
<line x1='12' x2='12' y1='22' y2='12' />
28+
<circle cx='18.5' cy='15.5' r='2.5' />
29+
<path d='M20.27 17.27 22 19' />
30+
</svg>`}
31+
/>
32+
33+
{/* MANUAL-CONTENT-START:intro */}
34+
Sim Studio's Knowledge Base is a powerful native feature that enables you to create, manage, and query custom knowledge bases directly within the platform. Using advanced AI embeddings and vector search technology, the Knowledge Base block allows you to build intelligent search capabilities into your workflows, making it easy to find and utilize relevant information across your organization.
35+
36+
The Knowledge Base system provides a comprehensive solution for managing organizational knowledge through its flexible and scalable architecture. With its built-in vector search capabilities, teams can perform semantic searches that understand meaning and context, going beyond traditional keyword matching.
37+
38+
Key features of the Knowledge Base include:
39+
40+
- Semantic Search: Advanced AI-powered search that understands meaning and context, not just keywords
41+
- Vector Embeddings: Automatic conversion of text into high-dimensional vectors for intelligent similarity matching
42+
- Custom Knowledge Bases: Create and manage multiple knowledge bases for different purposes or departments
43+
- Flexible Content Types: Support for various document formats and content types
44+
- Real-time Updates: Immediate indexing of new content for instant searchability
45+
46+
In Sim Studio, the Knowledge Base block enables your agents to perform intelligent semantic searches across your custom knowledge bases. This creates opportunities for automated information retrieval, content recommendations, and knowledge discovery as part of your AI workflows. The integration allows agents to search and retrieve relevant information programmatically, facilitating automated knowledge management tasks and ensuring that important information is easily accessible. By leveraging the Knowledge Base block, you can build intelligent agents that enhance information discovery while automating routine knowledge management tasks, improving team efficiency and ensuring consistent access to organizational knowledge.
47+
{/* MANUAL-CONTENT-END */}
48+
49+
## Usage Instructions
50+
51+
Perform semantic vector search across your knowledge base to find the most relevant content. Uses advanced AI embeddings to understand meaning and context, returning the most similar documents to your search query.
52+
53+
54+
55+
## Tools
56+
57+
### `knowledge_search`
58+
59+
Search for similar content in a knowledge base using vector similarity
60+
61+
#### Input
62+
63+
| Parameter | Type | Required | Description |
64+
| --------- | ---- | -------- | ----------- |
65+
| `knowledgeBaseId` | string | Yes | ID of the knowledge base to search in |
66+
| `query` | string | Yes | Search query text |
67+
| `topK` | number | No | Number of most similar results to return \(1-100\) |
68+
69+
#### Output
70+
71+
| Parameter | Type |
72+
| --------- | ---- |
73+
| `results` | string |
74+
| `query` | string |
75+
| `knowledgeBaseId` | string |
76+
| `topK` | string |
77+
| `totalResults` | string |
78+
| `message` | string |
79+
80+
81+
82+
## Block Configuration
83+
84+
### Input
85+
86+
| Parameter | Type | Required | Description |
87+
| --------- | ---- | -------- | ----------- |
88+
| `knowledgeBaseId` | string | Yes | Knowledge Base - Select knowledge base |
89+
90+
91+
92+
### Outputs
93+
94+
| Output | Type | Description |
95+
| ------ | ---- | ----------- |
96+
| `response` | object | Output from response |
97+
| ↳ `results` | json | results of the response |
98+
| ↳ `query` | string | query of the response |
99+
| ↳ `knowledgeBaseId` | string | knowledgeBaseId of the response |
100+
| ↳ `topK` | number | topK of the response |
101+
| ↳ `totalResults` | number | totalResults of the response |
102+
| ↳ `message` | string | message of the response |
103+
104+
105+
## Notes
106+
107+
- Category: `blocks`
108+
- Type: `knowledge`

‎apps/docs/content/docs/tools/meta.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@
2121
"image_generator",
2222
"jina",
2323
"jira",
24+
"knowledge",
2425
"linear",
2526
"linkup",
2627
"mem0",

‎apps/docs/content/docs/tools/microsoft_teams.mdx‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -64,6 +64,9 @@ import { BlockInfoCard } from "@/components/ui/block-info-card"
6464
<stop offset='0' stopColor='#5a62c3' />
6565
<stop offset='.5' stopColor='#4d55bd' />
6666
<stop offset='1' stopColor='#3940ab' />
67+
<stop offset='0' stopColor='#5a62c3' />
68+
<stop offset='.5' stopColor='#4d55bd' />
69+
<stop offset='1' stopColor='#3940ab' />
6770
</linearGradient>
6871
<path
6972
fill='url(#a)'

‎apps/docs/content/docs/tools/slack.mdx‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: Slack
3-
description: Send a message to Slack
3+
description: Send messages to Slack
44
---
55

66
import { BlockInfoCard } from "@/components/ui/block-info-card"
@@ -56,23 +56,24 @@ In Sim Studio, the Slack integration enables your agents to programmatically sen
5656

5757
## Usage Instructions
5858

59-
Send messages to any Slack channel using OAuth authentication. Integrate automated notifications and alerts into your workflow to keep your team informed.
59+
Comprehensive Slack integration with OAuth authentication. Send formatted messages using Slack
6060

6161

6262

6363
## Tools
6464

6565
### `slack_message`
6666

67-
Send messages to Slack channels or users through the Slack API. Enables direct communication and notifications with timestamp tracking and channel confirmation.
67+
Send messages to Slack channels or users through the Slack API. Supports Slack mrkdwn formatting.
6868

6969
#### Input
7070

7171
| Parameter | Type | Required | Description |
7272
| --------- | ---- | -------- | ----------- |
73-
| `apiKey` | string | Yes | Your Slack API token |
73+
| `botToken` | string | No | Bot token for Custom Bot |
74+
| `accessToken` | string | No | OAuth access token or bot token for Slack API |
7475
| `channel` | string | Yes | Target Slack channel \(e.g., #general\) |
75-
| `text` | string | Yes | Message text to send |
76+
| `text` | string | Yes | Message text to send \(supports Slack mrkdwn formatting\) |
7677

7778
#### Output
7879

@@ -89,7 +90,7 @@ Send messages to Slack channels or users through the Slack API. Enables direct c
8990

9091
| Parameter | Type | Required | Description |
9192
| --------- | ---- | -------- | ----------- |
92-
| `apiKey` | string | Yes | OAuth Token - Enter your Slack OAuth token |
93+
| `operation` | string | Yes | Operation |
9394

9495

9596

‎apps/sim/app/(landing)/contributors/page.tsx‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -433,7 +433,7 @@ export default function ContributorsPage() {
433433
<ResponsiveContainer width='100%' height={300} className='sm:!h-[400px]'>
434434
<BarChart
435435
data={filteredContributors?.slice(0, showAllContributors ? undefined : 10)}
436-
margin={{ top: 10, right: 5, bottom: 45, left: 5 }}
436+
margin={{ top: 10, right: 10, bottom: 60, left: 10 }}
437437
className='sm:!mx-2.5 sm:!mb-2.5'
438438
>
439439
<XAxis
@@ -448,11 +448,11 @@ export default function ContributorsPage() {
448448
return (
449449
<g transform={`translate(${x},${y})`}>
450450
<foreignObject
451-
x='-12'
452-
y='6'
453-
width='24'
454-
height='24'
455-
className='sm:!x-[-16] sm:!y-[8] sm:!w-[32] sm:!h-[32]'
451+
x='-16'
452+
y='8'
453+
width='32'
454+
height='32'
455+
style={{ overflow: 'visible' }}
456456
>
457457
<Avatar className='h-6 w-6 ring-1 ring-[#606060]/30 sm:h-8 sm:w-8'>
458458
<AvatarImage src={contributor?.avatar_url} />
@@ -464,8 +464,8 @@ export default function ContributorsPage() {
464464
</g>
465465
)
466466
}}
467-
height={50}
468-
className='sm:!h-[60px] text-neutral-400'
467+
height={60}
468+
className='text-neutral-400'
469469
/>
470470
<YAxis
471471
stroke='currentColor'

‎apps/sim/app/api/files/presigned/route.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -46,7 +46,7 @@ export async function POST(request: NextRequest) {
4646
Key: uniqueKey,
4747
ContentType: contentType,
4848
Metadata: {
49-
originalName: fileName,
49+
originalName: encodeURIComponent(fileName),
5050
uploadedAt: new Date().toISOString(),
5151
},
5252
})

‎apps/sim/app/api/knowledge/[id]/documents/[documentId]/chunks/[chunkId]/route.ts‎

Lines changed: 21 additions & 82 deletions
Original file line numberDiff line numberDiff line change
@@ -1,79 +1,21 @@
1-
import { and, eq, isNull } from 'drizzle-orm'
1+
import { eq } from 'drizzle-orm'
22
import { type NextRequest, NextResponse } from 'next/server'
33
import { z } from 'zod'
44
import { getSession } from '@/lib/auth'
55
import { createLogger } from '@/lib/logs/console-logger'
66
import { db } from '@/db'
7-
import { document, embedding, knowledgeBase } from '@/db/schema'
7+
import { embedding } from '@/db/schema'
8+
import { checkChunkAccess } from '../../../../../utils'
89

910
const logger = createLogger('ChunkByIdAPI')
1011

11-
// Schema for chunk updates
1212
const UpdateChunkSchema = z.object({
1313
content: z.string().min(1, 'Content is required').optional(),
1414
enabled: z.boolean().optional(),
1515
searchRank: z.number().min(0).optional(),
1616
qualityScore: z.number().min(0).max(1).optional(),
1717
})
1818

19-
async function checkChunkAccess(
20-
knowledgeBaseId: string,
21-
documentId: string,
22-
chunkId: string,
23-
userId: string
24-
) {
25-
// First check knowledge base access
26-
const kb = await db
27-
.select({
28-
id: knowledgeBase.id,
29-
userId: knowledgeBase.userId,
30-
})
31-
.from(knowledgeBase)
32-
.where(and(eq(knowledgeBase.id, knowledgeBaseId), isNull(knowledgeBase.deletedAt)))
33-
.limit(1)
34-
35-
if (kb.length === 0) {
36-
return { hasAccess: false, notFound: true, reason: 'Knowledge base not found' }
37-
}
38-
39-
const kbData = kb[0]
40-
41-
// Check if user owns the knowledge base
42-
if (kbData.userId !== userId) {
43-
return { hasAccess: false, reason: 'Unauthorized knowledge base access' }
44-
}
45-
46-
// Check if document exists and belongs to the knowledge base
47-
const doc = await db
48-
.select()
49-
.from(document)
50-
.where(
51-
and(
52-
eq(document.id, documentId),
53-
eq(document.knowledgeBaseId, knowledgeBaseId),
54-
isNull(document.deletedAt)
55-
)
56-
)
57-
.limit(1)
58-
59-
if (doc.length === 0) {
60-
return { hasAccess: false, notFound: true, reason: 'Document not found' }
61-
}
62-
63-
// Check if chunk exists and belongs to the document
64-
const chunk = await db
65-
.select()
66-
.from(embedding)
67-
.where(and(eq(embedding.id, chunkId), eq(embedding.documentId, documentId)))
68-
.limit(1)
69-
70-
if (chunk.length === 0) {
71-
return { hasAccess: false, notFound: true, reason: 'Chunk not found' }
72-
}
73-
74-
return { hasAccess: true, chunk: chunk[0], document: doc[0], knowledgeBase: kbData }
75-
}
76-
7719
export async function GET(
7820
req: NextRequest,
7921
{ params }: { params: Promise<{ id: string; documentId: string; chunkId: string }> }
@@ -95,14 +37,13 @@ export async function GET(
9537
session.user.id
9638
)
9739

98-
if (accessCheck.notFound) {
99-
logger.warn(
100-
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
101-
)
102-
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
103-
}
104-
10540
if (!accessCheck.hasAccess) {
41+
if (accessCheck.notFound) {
42+
logger.warn(
43+
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
44+
)
45+
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
46+
}
10647
logger.warn(
10748
`[${requestId}] User ${session.user.id} attempted unauthorized chunk access: ${accessCheck.reason}`
10849
)
@@ -144,14 +85,13 @@ export async function PUT(
14485
session.user.id
14586
)
14687

147-
if (accessCheck.notFound) {
148-
logger.warn(
149-
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
150-
)
151-
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
152-
}
153-
15488
if (!accessCheck.hasAccess) {
89+
if (accessCheck.notFound) {
90+
logger.warn(
91+
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
92+
)
93+
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
94+
}
15595
logger.warn(
15696
`[${requestId}] User ${session.user.id} attempted unauthorized chunk update: ${accessCheck.reason}`
15797
)
@@ -235,14 +175,13 @@ export async function DELETE(
235175
session.user.id
236176
)
237177

238-
if (accessCheck.notFound) {
239-
logger.warn(
240-
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
241-
)
242-
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
243-
}
244-
245178
if (!accessCheck.hasAccess) {
179+
if (accessCheck.notFound) {
180+
logger.warn(
181+
`[${requestId}] ${accessCheck.reason}: KB=${knowledgeBaseId}, Doc=${documentId}, Chunk=${chunkId}`
182+
)
183+
return NextResponse.json({ error: accessCheck.reason }, { status: 404 })
184+
}
246185
logger.warn(
247186
`[${requestId}] User ${session.user.id} attempted unauthorized chunk deletion: ${accessCheck.reason}`
248187
)

0 commit comments

Comments
 (0)