diff --git a/README.md b/README.md index 79ab9a50..01f77271 100644 --- a/README.md +++ b/README.md @@ -34,7 +34,7 @@ Without `KB_ROOT`, it opens a small sample folder. Any OpenAI-compatible endpoin - **Chat** answers from your files and links each source. Type @ to hand it a file from the library, set how long the model reasons with Effort, ask it to save a report or table as a PDF, Word or Excel file (with images from the library in a PDF or Word report, and justified text, landscape pages, a font, size, margins, line spacing, page breaks or a header and footer line when you ask for them; a report asked for in several formats is written once as Markdown and the others are made from it, and a later change is an edit to that file, with the PDF and Word documents made again), and have it move, rename, copy or delete library files (a delete waits for your approval). A chat goes into the library only when you save it, with **Save** or by asking the assistant. - **Search** finds file names, the words inside files, and, where it is on, meaning: passages that say the same thing in other words. It also reads the text and vectors GUFI's own extraction writes, in GUFI's common external database schema, in an index Studio did not build. Each library's search stops at a time limit, 20 seconds unless set longer in Settings for very large indexes, and a running search can be stopped; either way its query ends on the library's host too. -- **Library** browses and views documents, CSV tables, images, PDFs, office files, and 3D models, takes files and whole folders with their structure, and describes the folder you select: its size, contents, and labels. A library can name a second host over ssh that has its files, for an index whose own host does not. A folder on an SFTP or ssh host, a cluster or an rclone remote can be added as a copy that Studio keeps in step on this computer and indexes, which makes a host that allows SFTP only, such as an upload drop, searchable. A bucket of any S3 compatible store, on AWS or on premises, can be indexed through rclone from its listing alone, each object read only when opened (Preview). Libraries can be exported as a library list file that others import into their own Studio, from the Library's Add menu, in setup, or in Settings, so an administrator shares one setup, including an index built as root, and each person connects as themselves and sees what their own permissions allow. Where people sign in to a Studio beside a cluster, each person can have a space of their own there, written and indexed as their own account, and share folders in it with other accounts or groups through POSIX ACLs (Preview). +- **Library** browses and views documents, CSV tables, images, PDFs, office files, and 3D models, takes files and whole folders with their structure, uploads into a folder and downloads a file or a whole folder as a zip from its right-click menu, and describes the folder you select: its size, contents, and labels. A library can name a second host over ssh that has its files, for an index whose own host does not. A folder on an SFTP or ssh host, a cluster or an rclone remote can be added as a copy that Studio keeps in step on this computer and indexes, which makes a host that allows SFTP only, such as an upload drop, searchable. A bucket of any S3 compatible store, on AWS or on premises, can be indexed through rclone from its listing alone, each object read only when opened (Preview). Libraries can be exported as a library list file that others import into their own Studio, from the Library's Add menu, in setup, or in Settings, so an administrator shares one setup, including an index built as root, and each person connects as themselves and sees what their own permissions allow. Where people sign in to a Studio beside a cluster, each person can have a space of their own there, written and indexed as their own account, and share folders in it with other accounts or groups through POSIX ACLs (Preview). - **Transfers** copies, moves and syncs files between libraries, folders on this computer, your ACTIVATE clusters, saved ssh locations (including accounts that allow SFTP only), the remotes of your installed rclone (used with its existing configuration, which Studio never changes; sign-in off only) and CONDUIT, with a plan to confirm before each one starts and a list of every transfer and its progress. - **Query** answers structured questions about the files, such as the largest or newest, or runs read-only SQL. Query and search results export to CSV. - **Stats** maps each library, here or on another system: its folders by size, and its files by what they say. diff --git a/desktop/wails/bridge.go b/desktop/wails/bridge.go index 07319c4d..6a02d748 100644 --- a/desktop/wails/bridge.go +++ b/desktop/wails/bridge.go @@ -162,7 +162,10 @@ func (b *Bridge) SaveURL(path, name string) (string, error) { if !strings.HasPrefix(path, "/") || strings.HasPrefix(path, "//") { return "", errors.New("not a path on the Studio") } - res, err := (&http.Client{Timeout: 5 * time.Minute}).Get(fmt.Sprintf("http://127.0.0.1:%d%s", b.app.sup.Port(), path)) + // The Studio answers within minutes; a folder's zip is made as it is + // sent, so the whole transfer may take far longer. + client := &http.Client{Timeout: 2 * time.Hour, Transport: &http.Transport{ResponseHeaderTimeout: 5 * time.Minute}} + res, err := client.Get(fmt.Sprintf("http://127.0.0.1:%d%s", b.app.sup.Port(), path)) if err != nil { return "", err } diff --git a/docs/HELP.md b/docs/HELP.md index 3aed3c05..b3847b0c 100644 --- a/docs/HELP.md +++ b/docs/HELP.md @@ -63,6 +63,7 @@ The folder tree beside a viewer. Markdown, code, CSV and TSV tables, images, PDF - A library list file dropped onto the Library page is read as a list, not added as a file: the import opens with its libraries, and **Add the file to the library instead** adds it as a file. Studio checks the file with the server first, so any other `.json` file is added as before. This applies only where the arrow beside **+ Add** is shown, and not to files dropped from a file manager onto the Linux app's window, which are always added as files. - The eye button above the tree shows hidden files and folders, those whose names start with a dot, and remembers your choice. Tool folders such as `.git` and `.cache` stay out of the tree either way. - **Right-click** a file or folder to label it, add a folder, or delete it. In a read-only library the actions that would change it are shown disabled, with the reason on hover; labels still apply. +- **Right-click** a file and choose **Download** to save it to your computer without opening it. On a folder, **Download as zip** saves a zip of the files the Library shows in it, with its subfolders; files whose names start with a dot, and build and cache folders the Library leaves out, are left out of the zip too, and a link to somewhere outside the folder is not followed. A notice gives the number of files and the size as the download starts. A folder of more than 50,000 files or 8 GB (500 files or 1 GB in a library on another system, whose files are fetched one at a time) is refused with a notice, so download the folders inside it instead. **Upload files…** and **Upload a folder…** on a folder add files from your computer into that folder, a folder keeping its structure, with progress below the tree as for **+ Add**. In the app window the download goes to your Downloads folder. - An HTML page shows on its **Page** tab with its scripts running in a sandbox that reaches nothing outside the page. **Print**, in the viewer's header, opens it in a tab of its own with the print dialog, where Save as PDF is one of the destinations; a print button on the page itself works too. - **Find**, in the viewer's header, searches the open file. A file opened from a search result found by its words opens with Find set to those words; one found only by its meaning, name or folder opens without it. Punctuation inside a word matches any punctuation or space, as search reads it, so NIST_SP811 finds "NIST SP811". - **Indexed text** shows what search sees for a file, including text read from images. Text longer than about 60,000 characters, such as a long manual's, is shown as plain text so it opens quickly. diff --git a/docs/REFERENCE.md b/docs/REFERENCE.md index 1213ec78..3753483f 100644 --- a/docs/REFERENCE.md +++ b/docs/REFERENCE.md @@ -51,7 +51,7 @@ The details behind the user guide ([`HELP.md`](HELP.md)), by part of the app. Th - **Indexed text** shows exactly what the index holds for a file; for an image that is the text read from it plus a description written by a model. - **Find** highlights every match, stays in view while you scroll, and steps through matches with Enter, Shift+Enter, or its buttons. A file opened from a search result opens on its first match with the query already in the find bar; a PDF or office document opened that way opens on its indexed text, since page images cannot be searched. -- The right-click menu has Open, Labels…, and Delete for a file, and New folder inside…, Labels…, and Delete folder (with its contents, after a confirmation) for a folder. Delete removes the material and its index entries together. In a read-only library, New folder, Rename, Cut, Paste and Delete are disabled with the reason on hover, as are the selection's Move to… and Delete…, the viewer's Delete, and **+ Add**; Labels… is not. +- The right-click menu has Open, Download, Labels…, and Delete for a file, and New folder inside…, Upload files…, Upload a folder…, Download as zip, Labels…, and Delete folder (with its contents, after a confirmation) for a folder. Download as zip asks `GET /api/kb/download?path=…&folder=1&check=1` for the zip's name, file count and size first, then downloads `…&folder=1`; a folder over 50,000 files or 8 GB (500 files or 1 GB in a library on another system) is a 413. Without `folder=1` a folder is refused as before. Delete removes the material and its index entries together. In a read-only library, New folder, Upload files…, Upload a folder…, Rename, Cut, Paste and Delete are disabled with the reason on hover, as are the selection's Move to… and Delete…, the viewer's Delete, and **+ Add**; Labels… is not. - **Previews.** A PDF's **Pages** and an Office file's pages are images from `GET /api/kb/pdf-info` and `GET /api/kb/pdf-page`, which need poppler and, for Office files, LibreOffice; another library's are cached under its id. Without them, `GET /api/kb/office-preview?path=` reads a .pptx, .docx, .xlsx, .odp, .odt or .ods itself and answers `{ kind: "slides" | "document" | "sheets", ... }`: each slide's title and its paragraphs (with their level), tables and pictures; a document's headings, paragraphs, list items, tables and pictures; each sheet's first 200 rows and 30 columns. Pictures (PNG, JPEG, GIF, WebP, BMP) come from `GET /api/kb/office-media?path=&name=`, only for media members of the package. The viewer names the tab **Slides**, **Document** or **Sheets**. A file with neither (a 97-2003 one, or a page image that fails to load and no outline) loses the tab and opens on its indexed text with one line saying a preview needs LibreOffice. - With All libraries selected, a `DELETE /api/kb/file` goes to the file's own library (`library=`, the path inside it); the server also takes the combined path with `library=all`. An action on files that names `all` where one library is needed answers 400 saying it works in one library at a time. - Panes resize at their edges and collapse from their headers. diff --git a/server-go/internal/kb/folderzip.go b/server-go/internal/kb/folderzip.go new file mode 100644 index 00000000..13a7bce5 --- /dev/null +++ b/server-go/internal/kb/folderzip.go @@ -0,0 +1,197 @@ +package kb + +import ( + "archive/zip" + "bytes" + "io" + "io/fs" + "net/http" + "os" + "path" + "path/filepath" + "strings" + "time" +) + +// A folder of a library downloads as a zip of what the Library shows in +// it: entries hidden by name (a dot first) and the machinery it leaves out +// are left out of the zip too, and a symbolic link is not followed, so the +// zip holds only files under the folder. Its size is checked before the +// first byte is sent, so a folder over the limits is refused with a status +// and not cut off partway. + +const ( + // zipMaxFiles and zipMaxBytes bound a folder on this host. + zipMaxFiles = 50000 + zipMaxBytes = 8 << 30 + // zipMaxRemoteFiles and zipMaxRemoteBytes bound a folder of a library + // on another system, each of whose files is fetched in turn. + zipMaxRemoteFiles = 500 + zipMaxRemoteBytes = 1 << 30 +) + +// zipFile is a file going into a folder's zip: its name inside the zip, +// and where it is read from. +type zipFile struct { + name string + abs string // a file on this host + rel string // a file of a library elsewhere, by its path in the library + size int64 + mod time.Time +} + +// zipName is the zip's name for the folder at rel: the folder's name, or +// the library's for its top. +func zipName(lib Library, rel string) string { + base := path.Base(strings.Trim(filepath.ToSlash(rel), "/")) + if base == "." || base == "/" || base == "" { + base = lib.Label + } + if base == "" { + base = "library" + } + return base +} + +// localFolderFiles is what goes into the zip of the folder abs on this +// host, under name. +func localFolderFiles(abs, name string) ([]zipFile, error) { + var out []zipFile + var total int64 + err := filepath.WalkDir(abs, func(p string, d fs.DirEntry, err error) error { + if err != nil { + // A folder that cannot be read is left out, as the Library + // leaves out what it cannot list. + if d != nil && d.IsDir() && p != abs { + return fs.SkipDir + } + return nil + } + if p == abs { + return nil + } + if HiddenName(d.Name()) || Excluded(d.Name(), d.IsDir()) { + if d.IsDir() { + return fs.SkipDir + } + return nil + } + if d.IsDir() || !d.Type().IsRegular() { + return nil + } + info, err := d.Info() + if err != nil { + return nil + } + rel, _ := filepath.Rel(abs, p) + total += info.Size() + out = append(out, zipFile{name: name + "/" + filepath.ToSlash(rel), abs: p, size: info.Size(), mod: info.ModTime()}) + if len(out) > zipMaxFiles || total > zipMaxBytes { + return errTooBig + } + return nil + }) + return out, err +} + +var errTooBig = Error(413, "this folder is over %d files or %d GB, too large to download as a zip; download the folders inside it one at a time", zipMaxFiles, zipMaxBytes>>30) + +// remoteFolderFiles is what goes into the zip of the folder rel of a +// library on another system, from its index. +func (m *Module) remoteFolderFiles(r *http.Request, lib Library, rel, name string) ([]zipFile, error) { + var out []zipFile + var total int64 + var walk func(dir string) error + walk = func(dir string) error { + list, err := m.remoteTree(r, lib, dir, false) + if err != nil { + return err + } + for _, e := range list { + isDir := e.Type == "dir" + if HiddenName(e.Name) || Excluded(e.Name, isDir) { + continue + } + child := strings.TrimPrefix(path.Join(dir, e.Name), "/") + if isDir { + if err := walk(child); err != nil { + return err + } + continue + } + total += e.Size + inside := strings.TrimPrefix(strings.TrimPrefix(child, strings.Trim(rel, "/")), "/") + f := zipFile{name: name + "/" + inside, rel: child, size: e.Size} + if e.Mtime > 0 { + f.mod = time.Unix(e.Mtime, 0) + } + out = append(out, f) + if len(out) > zipMaxRemoteFiles || total > zipMaxRemoteBytes { + return Error(413, "this folder is over %d files or %d GB, too large to download as a zip from %s; download the folders inside it one at a time", zipMaxRemoteFiles, zipMaxRemoteBytes>>30, lib.Label) + } + } + return nil + } + if err := walk(strings.Trim(rel, "/")); err != nil { + return nil, err + } + return out, nil +} + +// zipOrSize answers what a folder's zip would hold, for check=1, which the +// Library asks before it starts the download, so a folder over the limits +// is refused with a notice and not saved as a file holding the error; and +// otherwise sends the zip. +func (m *Module) zipOrSize(w http.ResponseWriter, r *http.Request, lib Library, name string, files []zipFile) error { + if qs(r, "check") != "1" { + return m.sendZip(w, r, lib, name, files) + } + var bytes int64 + for _, f := range files { + bytes += f.size + } + writeJSON(w, 200, map[string]any{"name": name + ".zip", "files": len(files), "bytes": bytes}) + return nil +} + +// sendZip writes files as a zip named name.zip. Once the first byte is +// sent the status stands, so a file that cannot be read then is left out +// and the zip is finished without it. +func (m *Module) sendZip(w http.ResponseWriter, r *http.Request, lib Library, name string, files []zipFile) error { + w.Header().Set("Content-Type", "application/zip") + w.Header().Set("Content-Disposition", contentDisposition(name+".zip")) + w.WriteHeader(200) + z := zip.NewWriter(w) + for _, f := range files { + if r.Context().Err() != nil { + break + } + var src io.ReadCloser + if f.abs != "" { + fh, err := os.Open(f.abs) + if err != nil { + continue + } + src = fh + } else { + _, data, err := LibraryResolver.Download(r, lib, f.rel) + if err != nil { + continue + } + src = io.NopCloser(bytes.NewReader(data)) + } + hdr := &zip.FileHeader{Name: f.name, Method: zip.Deflate} + if !f.mod.IsZero() { + hdr.Modified = f.mod + } + dst, err := z.CreateHeader(hdr) + if err == nil { + _, err = io.Copy(dst, src) + } + src.Close() + if err != nil { + break + } + } + return z.Close() +} diff --git a/server-go/internal/kb/folderzip_test.go b/server-go/internal/kb/folderzip_test.go new file mode 100644 index 00000000..8b15b0d2 --- /dev/null +++ b/server-go/internal/kb/folderzip_test.go @@ -0,0 +1,155 @@ +package kb + +import ( + "archive/zip" + "bytes" + "io" + "log/slog" + "net/http" + "net/http/httptest" + "os" + "path/filepath" + "sort" + "strings" + "testing" + + "github.com/parallelworks/activate-studio/server-go/internal/app" + "github.com/parallelworks/activate-studio/server-go/internal/config" +) + +// zipStub is a library on another system with folders. +type zipStub struct { + remoteStub + dirs map[string][]Entry +} + +func (s *zipStub) Tree(_ *http.Request, _ Library, rel string) ([]Entry, error) { + return s.dirs[rel], nil +} + +func zipEntries(t *testing.T, body []byte) map[string]string { + t.Helper() + z, err := zip.NewReader(bytes.NewReader(body), int64(len(body))) + if err != nil { + t.Fatalf("not a zip: %v", err) + } + out := map[string]string{} + for _, f := range z.File { + rc, _ := f.Open() + b, _ := io.ReadAll(rc) + rc.Close() + out[f.Name] = string(b) + } + return out +} + +func names(m map[string]string) string { + var out []string + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return strings.Join(out, " ") +} + +func TestAFolderDownloadsAsAZip(t *testing.T) { + kbRoot, ix, outside := t.TempDir(), t.TempDir(), t.TempDir() + write := func(rel, s string) { + os.MkdirAll(filepath.Dir(filepath.Join(kbRoot, rel)), 0o755) + os.WriteFile(filepath.Join(kbRoot, rel), []byte(s), 0o644) + } + write("reports/plan.md", "# Plan") + write("reports/sub/notes.txt", "notes") + write("reports/.secret", "hidden") + write("reports/node_modules/x.js", "machinery") + write("reports/old.zip", "archive") + write("top.txt", "top") + os.WriteFile(filepath.Join(outside, "elsewhere.txt"), []byte("outside"), 0o644) + os.Symlink(outside, filepath.Join(kbRoot, "reports/linked-dir")) + os.Symlink(filepath.Join(outside, "elsewhere.txt"), filepath.Join(kbRoot, "reports/linked-file.txt")) + + env := map[string]string{"KB_ROOT": kbRoot, "INDEX_BASE": ix, "STUDIO_NODE": "off", "GUFI_BIN": filepath.Join(ix, "no-gufi")} + cfg, err := config.Load(func(k string) string { return env[k] }) + if err != nil { + t.Fatal(err) + } + m := New(app.New(cfg, slog.New(slog.DiscardHandler), nil, nil)) + mux := http.NewServeMux() + m.Routes(mux) + get := func(url string) *httptest.ResponseRecorder { + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, httptest.NewRequest("GET", url, nil)) + return rec + } + + // The folder, with what the Library shows in it and nothing a link + // points to outside it. + rec := get("/api/kb/download?path=reports&folder=1") + if rec.Code != 200 || rec.Header().Get("Content-Type") != "application/zip" || rec.Header().Get("Content-Disposition") != `attachment; filename="reports.zip"` { + t.Fatalf("%d %v %s", rec.Code, rec.Header(), rec.Body.String()) + } + got := zipEntries(t, rec.Body.Bytes()) + if names(got) != "reports/plan.md reports/sub/notes.txt" || got["reports/sub/notes.txt"] != "notes" { + t.Errorf("zip holds %q", names(got)) + } + // A folder inside a folder is named for itself. + if got := zipEntries(t, get("/api/kb/download?path=reports/sub&folder=1").Body.Bytes()); names(got) != "sub/notes.txt" { + t.Errorf("sub: %q", names(got)) + } + // Without folder=1 a folder is refused, as before. + if rec := get("/api/kb/download?path=reports"); rec.Code != 400 { + t.Errorf("unnamed folder: %d", rec.Code) + } + // A file still downloads as itself. + if rec := get("/api/kb/download?path=reports/plan.md"); rec.Code != 200 || rec.Body.String() != "# Plan" { + t.Errorf("file: %d %q", rec.Code, rec.Body.String()) + } + // A path that leaves the library is refused. + if rec := get("/api/kb/download?path=../&folder=1"); rec.Code != 400 { + t.Errorf("escape: %d", rec.Code) + } + + // A folder of a library on another system is read through it, file + // by file, when the Library names it a folder. + stub := &zipStub{remoteStub: remoteStub{files: map[string][]byte{"docs/one.txt": []byte("one"), "docs/deep/two.txt": []byte("two"), "docs/.hidden": []byte("h")}}, dirs: map[string][]Entry{ + "docs": {{Name: "one.txt", Type: "file", Size: 3}, {Name: "deep", Type: "dir"}, {Name: ".hidden", Type: "file"}, {Name: "build", Type: "dir"}}, + "docs/deep": {{Name: "two.txt", Type: "file", Size: 3, Mtime: 1700000000}}, + "docs/build": {{Name: "out.txt", Type: "file"}}, + }} + old := LibraryResolver + LibraryResolver = stub + defer func() { LibraryResolver = old }() + rec = get("/api/kb/download?library=far&path=docs&folder=1") + if rec.Code != 200 { + t.Fatalf("remote: %d %s", rec.Code, rec.Body.String()) + } + if got := zipEntries(t, rec.Body.Bytes()); names(got) != "docs/deep/two.txt docs/one.txt" || got["docs/deep/two.txt"] != "two" || stub.downloads != 2 { + t.Errorf("remote zip %q, %d downloads", names(got), stub.downloads) + } + // Too many files is refused before anything is sent. + many := []Entry{} + for i := 0; i <= zipMaxRemoteFiles; i++ { + many = append(many, Entry{Name: "f" + strings.Repeat("x", i%7) + string(rune('a'+i%26)) + ".txt", Type: "file"}) + } + stub.dirs["big"] = many + if rec := get("/api/kb/download?library=far&path=big&folder=1"); rec.Code != 413 || !strings.Contains(rec.Body.String(), "too large") { + t.Errorf("big: %d %s", rec.Code, rec.Body.String()) + } +} + +func TestAFolderZipIsSizedFirst(t *testing.T) { + kbRoot, ix := t.TempDir(), t.TempDir() + os.MkdirAll(filepath.Join(kbRoot, "data/deep"), 0o755) + os.WriteFile(filepath.Join(kbRoot, "data/a.txt"), []byte("12345"), 0o644) + os.WriteFile(filepath.Join(kbRoot, "data/deep/b.txt"), []byte("123"), 0o644) + env := map[string]string{"KB_ROOT": kbRoot, "INDEX_BASE": ix, "STUDIO_NODE": "off", "GUFI_BIN": filepath.Join(ix, "no-gufi")} + cfg, _ := config.Load(func(k string) string { return env[k] }) + m := New(app.New(cfg, slog.New(slog.DiscardHandler), nil, nil)) + mux := http.NewServeMux() + m.Routes(mux) + rec := httptest.NewRecorder() + mux.ServeHTTP(rec, httptest.NewRequest("GET", "/api/kb/download?path=data&folder=1&check=1", nil)) + if rec.Code != 200 || strings.TrimSpace(rec.Body.String()) != `{"bytes":8,"files":2,"name":"data.zip"}` { + t.Errorf("%d %s", rec.Code, rec.Body.String()) + } +} diff --git a/server-go/internal/kb/routes.go b/server-go/internal/kb/routes.go index a20ce109..47f09238 100644 --- a/server-go/internal/kb/routes.go +++ b/server-go/internal/kb/routes.go @@ -555,6 +555,16 @@ func (m *Module) download(w http.ResponseWriter, r *http.Request) error { if LibraryResolver == nil { return Error(409, "%s has no files on this host", lib.Label) } + // A folder, which the Library names with folder=1, comes down as a + // zip of its files, each read as a download of it is. + if qs(r, "folder") == "1" { + name := zipName(lib, sub) + files, err := m.remoteFolderFiles(r, lib, sub, name) + if err != nil { + return err + } + return m.zipOrSize(w, r, lib, name, files) + } name, data, err := LibraryResolver.Download(r, lib, sub) if err != nil { return err @@ -578,6 +588,16 @@ func (m *Module) download(w http.ResponseWriter, r *http.Request) error { if err != nil { return err } + // A folder comes down as a zip when the Library names it a folder; + // otherwise a folder is not a file to download. + if st, err := os.Lstat(abs); err == nil && st.IsDir() && qs(r, "folder") == "1" { + name := zipName(lib, sub) + files, err := localFolderFiles(abs, name) + if err != nil { + return err + } + return m.zipOrSize(w, r, lib, name, files) + } return sendFile(w, abs, map[string]string{"Content-Type": MimeFor(abs), "Content-Disposition": contentDisposition(filepath.Base(abs))}) } diff --git a/web/src/libraryDownload.ts b/web/src/libraryDownload.ts new file mode 100644 index 00000000..bcbf077e --- /dev/null +++ b/web/src/libraryDownload.ts @@ -0,0 +1,63 @@ +import { toast } from '@parallelworks/ui' +import { api } from './api' + +/** + * Download from the Library's right-click menu. A file downloads as itself. + * A folder downloads as a zip of what the Library shows in it, after the + * server has said what the zip holds, so a folder too large to zip, or one + * with no files, is a notice and not a saved file holding the error. + */ + +/** What a folder's zip holds, as the server counts it. */ +export interface FolderZip { name: string; files: number; bytes: number } + +function sizeText(n: number): string { + if (n < 1024) return `${n} B` + if (n < 1024 * 1024) return `${(n / 1024).toFixed(0)} KB` + if (n < 1024 ** 3) return `${(n / 1024 / 1024).toFixed(1)} MB` + return `${(n / 1024 ** 3).toFixed(2)} GB` +} + +/** The notice when a folder's zip starts. */ +export function zipNotice(z: FolderZip): string { + return `Downloading ${z.name}: ${z.files} ${z.files === 1 ? 'file' : 'files'}, ${sizeText(z.bytes)}.` +} + +/** Starts a download of a path on the Studio by a link with a download + * attribute. A browser keeps it with its downloads, the macOS window saves + * it to Downloads, and the Linux and Windows window fetches it into + * Downloads and reports where it went. */ +export function saveFromStudio(url: string, name: string): void { + const a = document.createElement('a') + a.href = url + a.download = name + document.body.appendChild(a) + a.click() + a.remove() +} + +/** Downloads a library file, or a folder as a zip. */ +export async function downloadPath(path: string, isDir: boolean): Promise { + const name = path.split('/').filter(Boolean).pop() || 'library' + if (!isDir) { + saveFromStudio(api.downloadUrl(path), name) + return + } + const url = `${api.downloadUrl(path)}&folder=1` + let zip: FolderZip + try { + const res = await fetch(`${url}&check=1`) + const body = await res.json().catch(() => null) as (FolderZip & { error?: string }) | null + if (!res.ok || !body) throw new Error(body?.error ?? `the Studio answered ${res.status}`) + zip = body + } catch (e) { + toast.error(`Could not download ${name}: ${(e as Error).message}`) + return + } + if (!zip.files) { + toast.error(`${name} has no files to download.`) + return + } + saveFromStudio(url, zip.name) + toast.success(zipNotice(zip)) +} diff --git a/web/src/views/LibraryView.tsx b/web/src/views/LibraryView.tsx index 10c0affc..b9dcc469 100644 --- a/web/src/views/LibraryView.tsx +++ b/web/src/views/LibraryView.tsx @@ -1,7 +1,7 @@ import { ALL_LIBRARIES, LibrarySwitch, libraryWritable } from '../components/LibrarySwitch' import { currentLibrary, onLibraryChange } from '../api' // eslint-disable-next-line @typescript-eslint/no-unused-vars -import { useCallback, useEffect, useMemo, useRef, useState } from 'react' +import { useCallback, useEffect, useMemo, useRef, useState, type RefObject } from 'react' import { Button, Callout, Checkbox, ConfirmModal, EmptyState, IconButton, Input, Modal, ProgressBar, RefreshButton, toast, Tooltip, useContextMenu, type MenuEntry, @@ -16,8 +16,9 @@ import { AddLibraryDialog, ImportLibrariesDialog, useLibraryAddOffer, type Dropp import { checkList, LIST_FORMAT } from '../libraryLists' import { TagMenu } from '../components/TagMenu' import { sectionAllowed, useAppConfig } from '../config' -import { collectDropped, collectListed, type UploadItem } from '../upload' -import { onShellDrop } from '../nativeShell' +import { collectChosen, collectDropped, collectListed, type UploadItem } from '../upload' +import { chooseUpload, onShellDrop, shellListsFolders } from '../nativeShell' +import { downloadPath } from '../libraryDownload' import { startPathJob, startUpload } from '../transfers' import { TransferPanel } from '../components/TransferPanel' import { selectLibraryEntry } from '../components/LibraryDetails' @@ -70,6 +71,11 @@ export function LibraryView({ display, onDisplay }: { /** A menu choice that opens a dialog, run once the menu has closed and * handed focus back, so the dialog keeps the focus it takes. */ const afterMenu = useRef<(() => void) | null>(null) + // Upload files and Upload a folder in the right-click menu: the folder + // they go into, and the choosers they open. + const uploadDir = useRef('') + const filesInput = useRef(null) + const folderInput = useRef(null) /** Cut or copied paths, waiting for a destination. */ const [clipboard, setClipboard] = useState<{ mode: 'cut' | 'copy'; paths: string[] } | null>(null) const selPaths = useMemo(() => [...multiSel], [multiSel]) @@ -327,6 +333,25 @@ export function LibraryView({ display, onDisplay }: { if (isDir) { items.push({ label: 'New folder', disabled: !!locked, disabledReason: locked ?? undefined, onClick: later(() => setNameDialog({ mode: 'mkdir', path, isDir, value: '', busy: false, error: '' })) }) } + // With All libraries the top of the tree is no folder: it holds the + // libraries, so nothing is uploaded to it or downloaded from it. + const allTop = inAll && !path + if (isDir && !allTop) { + // The choosers open from the click itself, which a browser requires. + const choose = (input: RefObject) => () => { uploadDir.current = path; input.current?.click() } + items.push( + { label: 'Upload files…', disabled: !!locked, disabledReason: locked ?? undefined, onClick: choose(filesInput) }, + { + label: 'Upload a folder…', disabled: !!locked, disabledReason: locked ?? undefined, + // The Linux and Windows window's chooser cannot pick a folder, so + // the window lists one itself. + onClick: shellListsFolders() + ? () => void chooseUpload('Choose a folder to upload').then(files => uploadItems(path, collectListed(files))) + : choose(folderInput), + }, + ) + } + if (!allTop) items.push({ label: isDir ? 'Download as zip' : 'Download', onClick: () => void downloadPath(path, isDir) }) if (path && !libraryTop) { items.push({ label: 'Rename…', disabled: !!locked, disabledReason: locked ?? undefined, onClick: later(() => setNameDialog({ mode: 'rename', path, isDir, value: path.split('/').pop() ?? '', busy: false, error: '' })) }) } @@ -612,6 +637,15 @@ export function LibraryView({ display, onDisplay }: { {contextMenu} + { void uploadItems(uploadDir.current, collectChosen([...(e.target.files ?? [])])); e.target.value = '' }} + /> + { folderInput.current = el; el?.setAttribute('webkitdirectory', '') }} + type="file" hidden aria-hidden="true" tabIndex={-1} + onChange={e => { void uploadItems(uploadDir.current, collectChosen([...(e.target.files ?? [])])); e.target.value = '' }} + /> {shareTarget && ( setShareTarget(null)} library={shareTarget.library} path={shareTarget.path} /> )} diff --git a/web/test/libraryDownload.test.ts b/web/test/libraryDownload.test.ts new file mode 100644 index 00000000..e48d630b --- /dev/null +++ b/web/test/libraryDownload.test.ts @@ -0,0 +1,55 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import * as ui from '@parallelworks/ui' +import { downloadPath, zipNotice } from '../src/libraryDownload' + +let success: ReturnType +let error: ReturnType +let clicked: { href: string; download: string }[] +beforeEach(() => { + success = vi.spyOn(ui.toast, 'success').mockImplementation(() => '' as never) + error = vi.spyOn(ui.toast, 'error').mockImplementation(() => '' as never) + clicked = [] + // jsdom does not navigate; record each link the page clicks instead. + vi.spyOn(HTMLAnchorElement.prototype, 'click').mockImplementation(function (this: HTMLAnchorElement) { + clicked.push({ href: this.getAttribute('href') ?? '', download: this.download }) + }) +}) +afterEach(() => vi.restoreAllMocks()) + +const answer = (status: number, body: unknown) => + vi.spyOn(globalThis, 'fetch').mockResolvedValue(new Response(JSON.stringify(body), { status, headers: { 'Content-Type': 'application/json' } })) + +describe('Download from the right-click menu', () => { + it('downloads a file as itself, with no check first', async () => { + const f = answer(200, {}) + await downloadPath('reports/plan.md', false) + expect(f).not.toHaveBeenCalled() + expect(clicked).toEqual([{ href: '/api/kb/download?path=reports%2Fplan.md', download: 'plan.md' }]) + }) + + it('sizes a folder, then downloads it as a zip', async () => { + const f = answer(200, { name: 'reports.zip', files: 12, bytes: 3_500_000 }) + await downloadPath('reports', true) + expect(f).toHaveBeenCalledWith('/api/kb/download?path=reports&folder=1&check=1') + expect(clicked).toEqual([{ href: '/api/kb/download?path=reports&folder=1', download: 'reports.zip' }]) + expect(success).toHaveBeenCalledWith('Downloading reports.zip: 12 files, 3.3 MB.') + }) + + it('says why a folder cannot be zipped, and saves nothing', async () => { + answer(413, { error: 'this folder is over 50000 files or 8 GB, too large to download as a zip; download the folders inside it one at a time' }) + await downloadPath('archive', true) + expect(clicked).toEqual([]) + expect(error).toHaveBeenCalledWith('Could not download archive: this folder is over 50000 files or 8 GB, too large to download as a zip; download the folders inside it one at a time') + }) + + it('says when a folder has nothing to download', async () => { + answer(200, { name: 'empty.zip', files: 0, bytes: 0 }) + await downloadPath('empty', true) + expect(clicked).toEqual([]) + expect(error).toHaveBeenCalledWith('empty has no files to download.') + }) + + it('counts one file as one file', () => { + expect(zipNotice({ name: 'a.zip', files: 1, bytes: 900 })).toBe('Downloading a.zip: 1 file, 900 B.') + }) +})