Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ Without `KB_ROOT`, it opens a small sample folder. Any OpenAI-compatible endpoin

- **Chat** answers from your files and links each source. Type @ to hand it a file from the library, set how long the model reasons with Effort, ask it to save a report or table as a PDF, Word or Excel file (with images from the library in a PDF or Word report, and justified text, landscape pages, a font, size, margins, line spacing, page breaks or a header and footer line when you ask for them; a report asked for in several formats is written once as Markdown and the others are made from it, and a later change is an edit to that file, with the PDF and Word documents made again), and have it move, rename, copy or delete library files (a delete waits for your approval). A chat goes into the library only when you save it, with **Save** or by asking the assistant.
- **Search** finds file names, the words inside files, and, where it is on, meaning: passages that say the same thing in other words. It also reads the text and vectors GUFI's own extraction writes, in GUFI's common external database schema, in an index Studio did not build. Each library's search stops at a time limit, 20 seconds unless set longer in Settings for very large indexes, and a running search can be stopped; either way its query ends on the library's host too.
- **Library** browses and views documents, CSV tables, images, PDFs, office files, and 3D models, takes files and whole folders with their structure, and describes the folder you select: its size, contents, and labels. A library can name a second host over ssh that has its files, for an index whose own host does not. A folder on an SFTP or ssh host, a cluster or an rclone remote can be added as a copy that Studio keeps in step on this computer and indexes, which makes a host that allows SFTP only, such as an upload drop, searchable. A bucket of any S3 compatible store, on AWS or on premises, can be indexed through rclone from its listing alone, each object read only when opened (Preview). Libraries can be exported as a library list file that others import into their own Studio, from the Library's Add menu, in setup, or in Settings, so an administrator shares one setup, including an index built as root, and each person connects as themselves and sees what their own permissions allow. Where people sign in to a Studio beside a cluster, each person can have a space of their own there, written and indexed as their own account, and share folders in it with other accounts or groups through POSIX ACLs (Preview).
- **Library** browses and views documents, CSV tables, images, PDFs, office files, and 3D models, takes files and whole folders with their structure, uploads into a folder and downloads a file or a whole folder as a zip from its right-click menu, and describes the folder you select: its size, contents, and labels. A library can name a second host over ssh that has its files, for an index whose own host does not. A folder on an SFTP or ssh host, a cluster or an rclone remote can be added as a copy that Studio keeps in step on this computer and indexes, which makes a host that allows SFTP only, such as an upload drop, searchable. A bucket of any S3 compatible store, on AWS or on premises, can be indexed through rclone from its listing alone, each object read only when opened (Preview). Libraries can be exported as a library list file that others import into their own Studio, from the Library's Add menu, in setup, or in Settings, so an administrator shares one setup, including an index built as root, and each person connects as themselves and sees what their own permissions allow. Where people sign in to a Studio beside a cluster, each person can have a space of their own there, written and indexed as their own account, and share folders in it with other accounts or groups through POSIX ACLs (Preview).
- **Transfers** copies, moves and syncs files between libraries, folders on this computer, your ACTIVATE clusters, saved ssh locations (including accounts that allow SFTP only), the remotes of your installed rclone (used with its existing configuration, which Studio never changes; sign-in off only) and CONDUIT, with a plan to confirm before each one starts and a list of every transfer and its progress.
- **Query** answers structured questions about the files, such as the largest or newest, or runs read-only SQL. Query and search results export to CSV.
- **Stats** maps each library, here or on another system: its folders by size, and its files by what they say.
Expand Down
5 changes: 4 additions & 1 deletion desktop/wails/bridge.go
Original file line number Diff line number Diff line change
Expand Up @@ -162,7 +162,10 @@ func (b *Bridge) SaveURL(path, name string) (string, error) {
if !strings.HasPrefix(path, "/") || strings.HasPrefix(path, "//") {
return "", errors.New("not a path on the Studio")
}
res, err := (&http.Client{Timeout: 5 * time.Minute}).Get(fmt.Sprintf("http://127.0.0.1:%d%s", b.app.sup.Port(), path))
// The Studio answers within minutes; a folder's zip is made as it is
// sent, so the whole transfer may take far longer.
client := &http.Client{Timeout: 2 * time.Hour, Transport: &http.Transport{ResponseHeaderTimeout: 5 * time.Minute}}
res, err := client.Get(fmt.Sprintf("http://127.0.0.1:%d%s", b.app.sup.Port(), path))
if err != nil {
return "", err
}
Expand Down
1 change: 1 addition & 0 deletions docs/HELP.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,7 @@ The folder tree beside a viewer. Markdown, code, CSV and TSV tables, images, PDF
- A library list file dropped onto the Library page is read as a list, not added as a file: the import opens with its libraries, and **Add the file to the library instead** adds it as a file. Studio checks the file with the server first, so any other `.json` file is added as before. This applies only where the arrow beside **+ Add** is shown, and not to files dropped from a file manager onto the Linux app's window, which are always added as files.
- The eye button above the tree shows hidden files and folders, those whose names start with a dot, and remembers your choice. Tool folders such as `.git` and `.cache` stay out of the tree either way.
- **Right-click** a file or folder to label it, add a folder, or delete it. In a read-only library the actions that would change it are shown disabled, with the reason on hover; labels still apply.
- **Right-click** a file and choose **Download** to save it to your computer without opening it. On a folder, **Download as zip** saves a zip of the files the Library shows in it, with its subfolders; files whose names start with a dot, and build and cache folders the Library leaves out, are left out of the zip too, and a link to somewhere outside the folder is not followed. A notice gives the number of files and the size as the download starts. A folder of more than 50,000 files or 8 GB (500 files or 1 GB in a library on another system, whose files are fetched one at a time) is refused with a notice, so download the folders inside it instead. **Upload files…** and **Upload a folder…** on a folder add files from your computer into that folder, a folder keeping its structure, with progress below the tree as for **+ Add**. In the app window the download goes to your Downloads folder.
- An HTML page shows on its **Page** tab with its scripts running in a sandbox that reaches nothing outside the page. **Print**, in the viewer's header, opens it in a tab of its own with the print dialog, where Save as PDF is one of the destinations; a print button on the page itself works too.
- **Find**, in the viewer's header, searches the open file. A file opened from a search result found by its words opens with Find set to those words; one found only by its meaning, name or folder opens without it. Punctuation inside a word matches any punctuation or space, as search reads it, so NIST_SP811 finds "NIST SP811".
- **Indexed text** shows what search sees for a file, including text read from images. Text longer than about 60,000 characters, such as a long manual's, is shown as plain text so it opens quickly.
Expand Down
2 changes: 1 addition & 1 deletion docs/REFERENCE.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ The details behind the user guide ([`HELP.md`](HELP.md)), by part of the app. Th

- **Indexed text** shows exactly what the index holds for a file; for an image that is the text read from it plus a description written by a model.
- **Find** highlights every match, stays in view while you scroll, and steps through matches with Enter, Shift+Enter, or its buttons. A file opened from a search result opens on its first match with the query already in the find bar; a PDF or office document opened that way opens on its indexed text, since page images cannot be searched.
- The right-click menu has Open, Labels…, and Delete for a file, and New folder inside…, Labels…, and Delete folder (with its contents, after a confirmation) for a folder. Delete removes the material and its index entries together. In a read-only library, New folder, Rename, Cut, Paste and Delete are disabled with the reason on hover, as are the selection's Move to… and Delete…, the viewer's Delete, and **+ Add**; Labels… is not.
- The right-click menu has Open, Download, Labels…, and Delete for a file, and New folder inside…, Upload files…, Upload a folder…, Download as zip, Labels…, and Delete folder (with its contents, after a confirmation) for a folder. Download as zip asks `GET /api/kb/download?path=…&folder=1&check=1` for the zip's name, file count and size first, then downloads `…&folder=1`; a folder over 50,000 files or 8 GB (500 files or 1 GB in a library on another system) is a 413. Without `folder=1` a folder is refused as before. Delete removes the material and its index entries together. In a read-only library, New folder, Upload files…, Upload a folder…, Rename, Cut, Paste and Delete are disabled with the reason on hover, as are the selection's Move to… and Delete…, the viewer's Delete, and **+ Add**; Labels… is not.
- **Previews.** A PDF's **Pages** and an Office file's pages are images from `GET /api/kb/pdf-info` and `GET /api/kb/pdf-page`, which need poppler and, for Office files, LibreOffice; another library's are cached under its id. Without them, `GET /api/kb/office-preview?path=` reads a .pptx, .docx, .xlsx, .odp, .odt or .ods itself and answers `{ kind: "slides" | "document" | "sheets", ... }`: each slide's title and its paragraphs (with their level), tables and pictures; a document's headings, paragraphs, list items, tables and pictures; each sheet's first 200 rows and 30 columns. Pictures (PNG, JPEG, GIF, WebP, BMP) come from `GET /api/kb/office-media?path=&name=<member>`, only for media members of the package. The viewer names the tab **Slides**, **Document** or **Sheets**. A file with neither (a 97-2003 one, or a page image that fails to load and no outline) loses the tab and opens on its indexed text with one line saying a preview needs LibreOffice.
- With All libraries selected, a `DELETE /api/kb/file` goes to the file's own library (`library=<id>`, the path inside it); the server also takes the combined path with `library=all`. An action on files that names `all` where one library is needed answers 400 saying it works in one library at a time.
- Panes resize at their edges and collapse from their headers.
Expand Down
197 changes: 197 additions & 0 deletions server-go/internal/kb/folderzip.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,197 @@
package kb

import (
"archive/zip"
"bytes"
"io"
"io/fs"
"net/http"
"os"
"path"
"path/filepath"
"strings"
"time"
)

// A folder of a library downloads as a zip of what the Library shows in
// it: entries hidden by name (a dot first) and the machinery it leaves out
// are left out of the zip too, and a symbolic link is not followed, so the
// zip holds only files under the folder. Its size is checked before the
// first byte is sent, so a folder over the limits is refused with a status
// and not cut off partway.

const (
// zipMaxFiles and zipMaxBytes bound a folder on this host.
zipMaxFiles = 50000
zipMaxBytes = 8 << 30
// zipMaxRemoteFiles and zipMaxRemoteBytes bound a folder of a library
// on another system, each of whose files is fetched in turn.
zipMaxRemoteFiles = 500
zipMaxRemoteBytes = 1 << 30
)

// zipFile is a file going into a folder's zip: its name inside the zip,
// and where it is read from.
type zipFile struct {
name string
abs string // a file on this host
rel string // a file of a library elsewhere, by its path in the library
size int64
mod time.Time
}

// zipName is the zip's name for the folder at rel: the folder's name, or
// the library's for its top.
func zipName(lib Library, rel string) string {
base := path.Base(strings.Trim(filepath.ToSlash(rel), "/"))
if base == "." || base == "/" || base == "" {
base = lib.Label
}
if base == "" {
base = "library"
}
return base
}

// localFolderFiles is what goes into the zip of the folder abs on this
// host, under name.
func localFolderFiles(abs, name string) ([]zipFile, error) {
var out []zipFile
var total int64
err := filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error {
if err != nil {
// A folder that cannot be read is left out, as the Library
// leaves out what it cannot list.
if d != nil && d.IsDir() && p != abs {
return fs.SkipDir
}
return nil
}
if p == abs {
return nil
}
if HiddenName(d.Name()) || Excluded(d.Name(), d.IsDir()) {
if d.IsDir() {
return fs.SkipDir
}
return nil
}
if d.IsDir() || !d.Type().IsRegular() {
return nil
}
info, err := d.Info()
if err != nil {
return nil
}
rel, _ := filepath.Rel(abs, p)
total += info.Size()
out = append(out, zipFile{name: name + "/" + filepath.ToSlash(rel), abs: p, size: info.Size(), mod: info.ModTime()})
if len(out) > zipMaxFiles || total > zipMaxBytes {
return errTooBig
}
return nil
})
return out, err
}

var errTooBig = Error(413, "this folder is over %d files or %d GB, too large to download as a zip; download the folders inside it one at a time", zipMaxFiles, zipMaxBytes>>30)

// remoteFolderFiles is what goes into the zip of the folder rel of a
// library on another system, from its index.
func (m *Module) remoteFolderFiles(r *http.Request, lib Library, rel, name string) ([]zipFile, error) {
var out []zipFile
var total int64
var walk func(dir string) error
walk = func(dir string) error {
list, err := m.remoteTree(r, lib, dir, false)
if err != nil {
return err
}
for _, e := range list {
isDir := e.Type == "dir"
if HiddenName(e.Name) || Excluded(e.Name, isDir) {
continue
}
child := strings.TrimPrefix(path.Join(dir, e.Name), "/")
if isDir {
if err := walk(child); err != nil {
return err
}
continue
}
total += e.Size
inside := strings.TrimPrefix(strings.TrimPrefix(child, strings.Trim(rel, "/")), "/")
f := zipFile{name: name + "/" + inside, rel: child, size: e.Size}
if e.Mtime > 0 {
f.mod = time.Unix(e.Mtime, 0)
}
out = append(out, f)
if len(out) > zipMaxRemoteFiles || total > zipMaxRemoteBytes {
return Error(413, "this folder is over %d files or %d GB, too large to download as a zip from %s; download the folders inside it one at a time", zipMaxRemoteFiles, zipMaxRemoteBytes>>30, lib.Label)
}
}
return nil
}
if err := walk(strings.Trim(rel, "/")); err != nil {
return nil, err
}
return out, nil
}

// zipOrSize answers what a folder's zip would hold, for check=1, which the
// Library asks before it starts the download, so a folder over the limits
// is refused with a notice and not saved as a file holding the error; and
// otherwise sends the zip.
func (m *Module) zipOrSize(w http.ResponseWriter, r *http.Request, lib Library, name string, files []zipFile) error {
if qs(r, "check") != "1" {
return m.sendZip(w, r, lib, name, files)
}
var bytes int64
for _, f := range files {
bytes += f.size
}
writeJSON(w, 200, map[string]any{"name": name + ".zip", "files": len(files), "bytes": bytes})
return nil
}

// sendZip writes files as a zip named name.zip. Once the first byte is
// sent the status stands, so a file that cannot be read then is left out
// and the zip is finished without it.
func (m *Module) sendZip(w http.ResponseWriter, r *http.Request, lib Library, name string, files []zipFile) error {
w.Header().Set("Content-Type", "application/zip")
w.Header().Set("Content-Disposition", contentDisposition(name+".zip"))
w.WriteHeader(200)
z := zip.NewWriter(w)
for _, f := range files {
if r.Context().Err() != nil {
break
}
var src io.ReadCloser
if f.abs != "" {
fh, err := os.Open(f.abs)
if err != nil {
continue
}
src = fh
} else {
_, data, err := LibraryResolver.Download(r, lib, f.rel)
if err != nil {
continue
}
src = io.NopCloser(bytes.NewReader(data))
}
hdr := &zip.FileHeader{Name: f.name, Method: zip.Deflate}
if !f.mod.IsZero() {
hdr.Modified = f.mod
}
dst, err := z.CreateHeader(hdr)
if err == nil {
_, err = io.Copy(dst, src)
}
src.Close()
if err != nil {
break
}
}
return z.Close()
}
Loading
Loading