Skip to content

feat(drivers): add Rakuten Drive support - #2084

Open
rufusu wants to merge 16 commits into
OpenListTeam:mainfrom
rufusu:feat/rakutenDrive
Open

rufusu wants to merge 16 commits into
OpenListTeam:mainfrom
rufusu:feat/rakutenDrive

Conversation

@rufusu

@rufusu rufusu commented Feb 6, 2026 •

Copy link
Copy Markdown

feat(rakuten_drive): add Rakuten Drive storage driver

Description / 描述

  • Added a new RakutenDrive driver with list, link, mkdir, move, rename, copy, delete, upload (S3 multipart), and storage quota support.
  • Implemented token refresh + JWT exp parsing, request wrapper, and path normalization utilities.
  • Registered the driver in drivers/all.go.

Motivation and Context / 背景

  • Enables OpenList to connect to 楽天ドライブ via its API for file operations and uploads.

How Has This Been Tested? / 测试

  • Built and ran the Docker container locally to verify the Rakuten Drive works.

Checklist / 检查清单

  • I have read the CONTRIBUTING document.
    我已阅读 CONTRIBUTING 文档。
  • I have formatted my code with go fmt or prettier.
    我已使用 go fmt 或 prettier 格式化提交的代码。
  • I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
    我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
  • I have requested review from relevant code authors using the "Request review" feature when applicable.
    我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
  • I have updated the repository accordingly (If it’s needed).
    我已相应更新了相关仓库(若适用)。
  • OpenList-Frontend #XXXX
  • OpenList-Docs #XXXX

@rufusu
rufusu marked this pull request as draft February 6, 2026 04:52
@rufusu
rufusu marked this pull request as ready for review February 6, 2026 04:53
@rufusu
rufusu marked this pull request as draft February 6, 2026 04:53
@rufusu
rufusu marked this pull request as ready for review February 6, 2026 05:08
Comment thread drivers/rakuten_drive/driver.go Outdated
if srcRemote == "" {
srcRemote = d.toRemotePath(srcObj.GetPath(), srcObj.IsDir())
}
prefix := path.Dir(srcRemote)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe you can extract it as a function? It was used four times.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Will update this to reduce duplication as well.

Comment thread drivers/rakuten_drive/types.go
Comment thread drivers/rakuten_drive/driver.go
Comment thread drivers/rakuten_drive/driver.go Outdated
}
if strings.HasSuffix(itemPath, "/") {
isFolder = true
}

@dezhishen dezhishen Feb 6, 2026 •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind moving this part into its own method? It would make the code easier to read and maintain.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion — I’ll update this to reduce duplication.

This comment was marked as duplicate.

@rufusu

rufusu commented Feb 7, 2026

Copy link
Copy Markdown
Author

refactor(rakuten_drive): extract repeated logic into helper methods

Description / 描述

  • Extracted normalizePrefix method to handle directory prefix normalization logic that was repeated in Move, Rename, Copy, and Remove methods.
  • Extracted detectIsFolder method to encapsulate folder detection logic from the parseList method.
  • Improved code maintainability and readability by reducing code duplication.

Motivation and Context / 背景

  • Addressed code review feedback requesting extraction of repeated logic into reusable methods.
  • The prefix normalization pattern was duplicated 4 times across different methods.
  • The folder detection logic in parseList was complex and warranted its own method for better clarity.

How Has This Been Tested? / 测试

  • Verified that the code compiles without errors.
  • Logic behavior remains unchanged, only code organization improved.

Checklist / 检查清单

  • I have read the CONTRIBUTING document.
    我已阅读 CONTRIBUTING 文档。
  • I have formatted my code with go fmt or prettier.
    我已使用 go fmt 或 prettier 格式化提交的代码。
  • I have added appropriate labels to this PR (or mentioned needed labels in the description if lacking permissions).
    我已为此 PR 添加了适当的标签(如无权限或需要的标签不存在,请在描述中说明,管理员将后续处理)。
  • I have requested review from relevant code authors using the "Request review" feature when applicable.
    我已在适当情况下使用"Request review"功能请求相关代码作者进行审查。
  • I have updated the repository accordingly (If it's needed).
    我已相应更新了相关仓库(若适用)。
  • OpenList-Frontend #XXXX
  • OpenList-Docs #XXXX

Comment thread drivers/rakuten_drive/driver.go Outdated
}
if strings.HasSuffix(itemPath, "/") {
isFolder = true
}

This comment was marked as duplicate.

Comment thread drivers/rakuten_drive/util.go Outdated
return time.Unix(exp, 0), nil
}

func (d *RakutenDrive) toRemotePath(p string, isDir bool) string {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Typically, we express conversions as "from A to B" (e.g., a2b), or retrievals as "get something" (e.g., getSomething). This method name fails to accurately convey its purpose.
toLocalPath has the same promble.

@xrgzs xrgzs added enhancement Module: Driver Driver-Related Issue/PR labels Feb 19, 2026
@Suyunmeng
Suyunmeng force-pushed the main branch 4 times, most recently from a31fd53 to 7bea29c Compare April 2, 2026 17:27
rufusu and others added 2 commits April 17, 2026 00:52
…2RemotePath/remotePath2LocalPath

Follow project naming conventions as per reviewer feedback

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new Rakuten Drive storage driver to the OpenList drivers set, enabling basic file operations and multipart uploads against Rakuten Drive’s API, and registers it for runtime availability.

Changes:

  • Introduces drivers/rakuten_drive with auth/token refresh, request helpers, path utilities, and API response types.
  • Implements core driver operations (list/link/mkdir/move/rename/copy/remove/upload/quota).
  • Registers the new driver via drivers/all.go.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 7 comments.

Show a summary per file
File Description
drivers/rakuten_drive/driver.go Implements RakutenDrive driver operations, including S3 multipart upload and quota retrieval.
drivers/rakuten_drive/util.go Adds token refresh/JWT exp parsing, request wrapper, and path/time normalization helpers.
drivers/rakuten_drive/types.go Defines API response structs and a File object wrapper carrying extra metadata.
drivers/rakuten_drive/meta.go Defines driver config/addition schema and registers the driver.
drivers/all.go Imports the driver package to register it in the global driver set.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +408 to +424
// 4) check status
checkBody := base.Json{"key": initResp.UploadID}
for i := 0; i < 30; i++ {
var checkResp uploadCheckResp
_, err = d.newForestRequest(ctx, http.MethodPost, forestBase+"/v3/files/check", checkBody, &checkResp)
if err != nil {
return nil, err
}
if strings.EqualFold(checkResp.State, "complete") {
break
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(1 * time.Second):
}
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Put, the files/check polling loop breaks when state == complete, but if it never becomes complete within 30 tries the code still proceeds to complete/upload and returns success. Consider tracking whether completion was reached and returning a timeout/error (or handling explicit failure states) before continuing.

Copilot uses AI. Check for mistakes.
Comment thread drivers/rakuten_drive/driver.go Outdated
Comment on lines +527 to +529
if localDir != "" && !strings.HasSuffix(localDir, "/") && !strings.HasPrefix(localPath, "/") {
localPath = path.Join(localDir, name)
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This localPath adjustment branch appears unreachable: remotePath2LocalPath always returns a string starting with "/" (either "/" or "/..."), so !strings.HasPrefix(localPath, "/") will never be true. Consider removing this conditional or reworking it if a different path format is expected.

Suggested change
if localDir != "" && !strings.HasSuffix(localDir, "/") && !strings.HasPrefix(localPath, "/") {
localPath = path.Join(localDir, name)
}

Copilot uses AI. Check for mistakes.
Comment thread drivers/rakuten_drive/driver.go Outdated
Comment on lines +128 to +130
if remoteDir == "/" {
remoteDir = ""
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

localPath2RemotePath(..., true) never returns "/" (it returns "" for root), so these remoteDir == "/" checks are dead code. Consider removing them to avoid confusion, or adjust localPath2RemotePath to return "/" for root consistently if that’s required by the API.

Suggested change
if remoteDir == "/" {
remoteDir = ""
}

Copilot uses AI. Check for mistakes.
Comment thread drivers/rakuten_drive/driver.go Outdated
Comment on lines +305 to +307
if remoteDir == "/" {
remoteDir = ""
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same as above: localPath2RemotePath(..., true) returns "" (not "/") for root, so this remoteDir == "/" branch is unreachable and can be removed (or make root handling consistent in one place).

Suggested change
if remoteDir == "/" {
remoteDir = ""
}

Copilot uses AI. Check for mistakes.
if result != nil {
req.SetResult(result)
}
return req.Execute(method, url)

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

newForestRequest returns req.Execute results without checking HTTP status codes. Resty does not treat 4xx/5xx as an error, so callers like MakeDir/Move/Rename/Remove/Put can incorrectly report success on API failures. Consider explicitly checking resp.IsError() / resp.StatusCode() >= 400 and returning an error (ideally including status + response body) before returning the response.

Suggested change
return req.Execute(method, url)
resp, err := req.Execute(method, url)
if err != nil {
return resp, err
}
if resp != nil && resp.IsError() {
return resp, fmt.Errorf("forest request %s %s failed with status %d: %s", method, url, resp.StatusCode(), strings.TrimSpace(resp.String()))
}
return resp, nil

Copilot uses AI. Check for mistakes.
Comment on lines +230 to +247
if resp.Key != "" {
checkBody := base.Json{"key": resp.Key}
for i := 0; i < 30; i++ {
var checkResp uploadCheckResp
_, err = d.newForestRequest(ctx, http.MethodPost, forestBase+"/v3/files/check", checkBody, &checkResp)
if err != nil {
return nil, err
}
if strings.EqualFold(checkResp.State, "complete") {
break
}
select {
case <-ctx.Done():
return nil, ctx.Err()
case <-time.After(1 * time.Second):
}
}
}

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Copy, the async completion polling loop breaks after 30s but still returns a successful model.Object even if the operation never reaches state == complete. This can surface as phantom copies. After the loop, verify completion and return a timeout/error when the state is still not complete.

Copilot uses AI. Check for mistakes.
Comment thread drivers/rakuten_drive/driver.go Outdated
Comment on lines +281 to +300
if resp.Key == "" {
return nil
}
checkBody := base.Json{"key": resp.Key}
for i := 0; i < 30; i++ {
var checkResp uploadCheckResp
_, err = d.newForestRequest(ctx, http.MethodPost, forestBase+"/v3/files/check", checkBody, &checkResp)
if err != nil {
return err
}
if strings.EqualFold(checkResp.State, "complete") {
return nil
}
select {
case <-ctx.Done():
return ctx.Err()
case <-time.After(1 * time.Second):
}
}
return nil

Copilot AI Apr 17, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove polls files/check up to 30 times but returns nil even if the operation never reaches state == complete (loop falls through). This can report a deletion as successful while it’s still pending/failed. Consider returning a timeout/error after the loop when completion wasn’t observed.

Copilot uses AI. Check for mistakes.

@pikachuren pikachuren left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🙏 感谢 @Lucaszhu2001 提交!
🤖 AI 自动审核声明:本评审报告由 AI 自动生成,当前使用 Claude Opus 5 模型进行分析。
⚠️ AI 分析结果仅供参考,可能存在误判或遗漏。如您发现任何问题或有不同意见,欢迎随时提出讨论和纠正。
⚠️ 重要提醒:即使 AI 评审认为代码质量良好且建议合并,最终是否合并仍需由项目维护者进行人工判定。项目维护者会综合考虑代码质量、项目规划、技术方向、团队资源等多方面因素做出决策。

🎯 结论

🔄 Request Changes — 驱动可用,但 JWT 解析与 STS 凭证处理有几处健壮性问题需要补齐

📖 概要

feat(drivers): add Rakuten Drive support · 新增乐天 Drive 驱动。
核心改动:新增 drivers/rakuten_drive 包,用 refresh_token 换取 idToken,通过 filelink/token 获取 STS 临时凭证后走 S3 上传下载。

🧭 整体方案

技术路线是「refresh_token → idToken(JWT)→ STS 临时凭证 → S3 操作」的三级鉴权链。实现里有几处做得不错:ensureAccessToken 用 tokenMu 保护并提前 1 分钟刷新;服务端返回新 refresh_token 时会持久化(op.MustSaveDriverStorage)以支持轮换;filelink/token 响应中三个凭证字段有非空校验。主要问题集中在 JWT 解析的容错上。

📊 变更统计

5 个文件(+923 / -0 行) | 功能 ⭐⭐⭐⭐ | 最小改动 ⭐⭐⭐⭐ | 前向兼容 ⭐⭐⭐⭐⭐ | 方案设计 ⭐⭐⭐

🚨 关键问题

P0(阻塞合并):无

P1(建议修复):

  • ⚠️ drivers/rakuten_drive/util.go:refreshToken — 当 parseJWTExp 解析失败时,代码把 d.accessTokenExpire 设为零值 time.Time{}。再看 ensureAccessToken 的判断条件 !d.accessTokenExpire.IsZero() && time.Now().Before(...):零值会导致该条件恒为 false,于是每一次 API 调用前都会重新走一遍 refreshToken。这不仅带来大量冗余请求,还可能触发乐天侧的风控限流。请问是否考虑解析失败时回退到一个保守的固定有效期呢?例如:
if exp, err := parseJWTExp(resp.IDToken); err == nil {
    d.accessTokenExpire = exp
} else {
    // 解析失败时回退到保守有效期,避免每次请求都重新登录
    d.accessTokenExpire = time.Now().Add(30 * time.Minute)
}
  • ⚠️ parseJWTExp 用 base64.RawURLEncoding 解码 JWT payload。JWT 规范虽规定用 unpadded base64url,但部分实现会带 = 填充,此时 RawURLEncoding 会解码失败并触发上述每次刷新的问题。是否考虑先 strings.TrimRight(parts[1], "=") 再解码,或对两种编码都做尝试?
  • ⚠️ getFilelinkToken 拿到的 STS 凭证(AccessKeyID / SecretAccessKey / SessionToken)看起来是每次操作都重新获取,未做缓存。若单次批量操作涉及多个文件,会产生大量 token 请求。请问这是刻意为之(凭证与具体路径绑定)还是可以按目录缓存呢?

P2(可选):

  • 💡 refreshToken 使用全局的 base.RestyClient 而其他请求用 d.client,两者配置可能不一致(超时、UA 等)。建议统一~
  • 💡 请求中硬编码了 Origin / Referer 为 https://www.rakuten-drive.com,属于模拟 Web 端的常见做法。建议加一行注释说明必要性,避免后续被误删~
  • 💡 UploadToken 字段的 help 写的是「if required」,语义比较模糊。是否能明确说明何种场景下需要填写、从哪里获取?
  • 💡 refreshToken() 方法名与 Addition.RefreshToken 字段名相近,阅读时容易混淆,是否考虑改为 doRefreshAccessToken() 之类更明确的命名?
  • 💡 建议为 parseJWTExp 补充单元测试,覆盖正常 JWT、带 padding、缺少 exp、格式非法四种情况~

🔐 安全审查

  • 审查方式:扫描命令执行、TLS 绕过、硬编码凭证、可疑外连、base64 使用场景
  • 安全评估:✅ 未发现恶意代码
  • 详细结论:base64 用于 JWT payload 解码,属正常用途;无命令执行、无 TLS 校验绕过、无硬编码密钥;外连目标为 api.rakuten-drive.com 与 forest.sendy.jp(乐天 Drive 的存储服务域名)。STS 凭证经 credentials.NewStaticCredentials 传给 AWS SDK,未做日志输出,处理方式正确。

📂 逐文件分析

drivers/rakuten_drive/util.go

改动意图:实现鉴权链与路径转换。
代码逻辑:ensureAccessToken 检查缓存 → refreshToken 换取 idToken → parseJWTExp 提取过期时间;newForestRequest 统一注入鉴权头。
问题分析:并发保护与 refresh_token 轮换持久化都正确;核心问题是 JWT 解析失败时的降级策略(P1)与 padding 兼容性(P1)。

drivers/rakuten_drive/driver.go

改动意图:实现驱动主体操作。
问题分析:STS 凭证的非空校验到位;建议评估凭证缓存(P1)。

✅ 待处理清单

  • [P1] parseJWTExp 失败时回退到保守有效期,避免每次请求都重新登录
  • [P1] 兼容带 = padding 的 JWT payload
  • [P1] 评估 STS 凭证是否可按目录缓存以减少请求
  • [P2] 统一 HTTP client 的使用
  • [P2] 为 UploadToken 补充明确的获取说明
  • [P2] 为 parseJWTExp 补充单元测试

🎯 结论:🔄 Request Changes — 鉴权链设计合理、无安全隐患,但 JWT 解析失败会导致每次请求重新登录,建议先修复。

rufusu and others added 2 commits September 18, 2026 00:42
…r handling

- Fall back to a conservative 30-minute validity when JWT exp parsing fails instead of leaving a zero expiry, which forced a token refresh on every request
- Accept base64-padded JWT payloads when decoding the exp claim
- Cache filelink STS credentials per directory until shortly before expiry
- Return an error on HTTP 4xx/5xx in newForestRequest and doRefreshAccessToken so API failures are no longer treated as success
- Extract waitCheckComplete helper and return a timeout error when files/check polling in Put/Copy/Remove never reaches the complete state
- Remove dead remoteDir == "/" checks and the unreachable localPath branch in parseList
- Unify HTTP client usage via newHTTPClient and rename refreshToken to doRefreshAccessToken to avoid confusion with the refresh_token field
- Clarify upload_token help text and document the Origin/Referer headers
- Add unit tests for parseJWTExp covering padded, missing-exp, and invalid inputs
@rufusu

rufusu commented Sep 17, 2026

Copy link
Copy Markdown
Author

Thanks for the thorough review! All issues have been fixed in commit 03c686f. Here is an item-by-item response:

P1 Fixes

1. parseJWTExp failure causing a token refresh on every request ✅

Fixed as suggested — on parse failure we now fall back to a conservative validity instead of leaving a zero value:

if exp, err := parseJWTExp(resp.IDToken); err == nil {
    d.accessTokenExpire = exp
} else {
    // fall back to a conservative validity so we don't refresh on every request
    d.accessTokenExpire = time.Now().Add(30 * time.Minute)
}

2. JWT payload compatibility with = padding ✅

We now strings.TrimRight(parts[1], "=") before decoding, so RawURLEncoding handles both padded and unpadded encodings.

3. STS credential caching ✅

Added getSTSCredentials() which caches credentials per directory: validity is taken from the response's Expiration field and dropped 1 minute before actual expiry; a 30-minute TTL fallback applies when Expiration is missing or nearly due. Batch uploads to the same directory no longer trigger repeated filelink/token requests.

P2 Fixes

4. Unified HTTP client ✅

Extracted newHTTPClient(); doRefreshAccessToken (also renamed from refreshToken per your suggestion, to avoid confusion with the Addition.RefreshToken field) now uses d.client, and an HTTP 4xx/5xx status check was added.

5. Clarified UploadToken help ✅

The help text now states where the value comes from (Rakuten Drive web app request headers) and when it can be left empty; the hardcoded Origin/Referer in newForestRequest also got a comment explaining they are required to mimic the web app.

6. Unit tests for parseJWTExp ✅

Added util_test.go covering five cases: valid JWT, padded payload, missing exp, no dot separator, and invalid base64 — 5/5 passing.

Additional fixes (from the earlier Copilot review)

  • newForestRequest now checks resp.IsError(), so 4xx/5xx responses are no longer silently treated as success
  • The three files/check polling loops were extracted into waitCheckComplete(), which returns an error (including the last observed state) on timeout, fixing potential phantom successes in Put/Copy/Remove
  • Removed the dead remoteDir == "/" checks and the unreachable localPath branch in parseList

Verified with go build ./..., go vet, and go test ./drivers/rakuten_drive/ — all passing. Please take another look when you have a chance, thank you!

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Multipart uploads can publish truncated data, and several returned objects are incompatible with subsequent cached mutations.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (4)

drivers/rakuten_drive/driver.go:145

  • This result is a plain model.Object, but Move, Rename, and Remove all call getFile and reject anything that is not *File. Because internal/op caches the result of MakeDir, a newly created directory cannot be removed or renamed until the directory is re-listed. Return a *File here (with the embedded object fields) so the metadata contract is preserved.
	newPath := path.Join(parentDir.GetPath(), dirName)
	return &model.Object{
		ID:       d.localPath2RemotePath(newPath, true),
		Path:     newPath,
		Name:     dirName,
		IsFolder: true,
		Modified: time.Now(),
	}, nil

drivers/rakuten_drive/driver.go:241

  • The copy result is also a plain model.Object, while this driver requires *File metadata for later Remove, Move, or Rename calls. internal/op places the returned copy directly into the destination cache, so those operations fail on the freshly copied object until a refresh. Return *File here instead.
	newPath := path.Join(dstDir.GetPath(), srcObj.GetName())
	return &model.Object{
		ID:       d.localPath2RemotePath(newPath, srcObj.IsDir()),
		Path:     newPath,
		Name:     srcObj.GetName(),
		Size:     srcObj.GetSize(),
		IsFolder: srcObj.IsDir(),
		Modified: time.Now(),
	}, nil

drivers/rakuten_drive/driver.go:392

  • Uploads return a plain model.Object, but the result is cached by internal/op and later mutation/deletion paths require *File metadata via getFile. Consequently, deleting or renaming an uploaded file can fail until the directory is re-listed. Return a *File (and retain any available version metadata) from this method.
	return &model.Object{
		ID:       initResp.File[0].Path,
		Path:     path.Join(dstDir.GetPath(), file.GetName()),
		Name:     file.GetName(),
		Size:     size,
		Modified: time.Now(),
	}, nil

drivers/rakuten_drive/driver.go:354

  • The cleanup request uses the caller's context, so if the upload failed because that context was canceled or its deadline expired, AbortMultipartUpload is canceled immediately as well. This leaves incomplete multipart uploads in the S3 bucket until the provider's lifecycle cleanup; use a non-canceled cleanup context (with a bounded timeout) for the abort.
		_, _ = s3Client.AbortMultipartUploadWithContext(ctx, &s3.AbortMultipartUploadInput{
			Bucket:   aws.String(initResp.Bucket),
			Key:      aws.String(objectKey),
			UploadId: aws.String(s3UploadID),
		})
  • Files reviewed: 6/6 changed files
  • Comments generated: 4
  • Review effort level: Lite

Comment thread drivers/rakuten_drive/driver.go
Comment thread drivers/rakuten_drive/driver.go Outdated
Comment thread drivers/rakuten_drive/driver.go
Comment thread drivers/rakuten_drive/util.go Outdated
…s and uploads

- Return *File from MakeDir, Copy, and Put so objects cached by internal/op keep the *File metadata contract required by getFile in Move/Rename/Remove, allowing freshly created/copied/uploaded entries to be mutated without a re-list
- Retain version_id and last_modified from the upload init response in the object returned by Put
- Derive a bounded, cancellation-proof context for AbortMultipartUpload so aborted uploads are still cleaned up when the caller's context was already canceled or expired
- Also abort the multipart upload when CompleteMultipartUpload fails
- Fail the multipart upload with a short-read error when the stream ends before the declared size instead of publishing truncated data
- Only strip the root prefix in remotePath2LocalPath on a path-segment boundary so sibling names that merely start with the root folder are not mangled
- Add unit tests for remotePath2LocalPath root boundary handling
…STS cache

- Retry a file API request once with a forced token refresh on HTTP 401, so a revoked access token no longer leaves the storage broken until it is reconfigured
- End list pagination based on the raw page size returned by the API instead of the count of parsed items, so entries that cannot be mapped to a path cannot end paging early and drop later pages
- Clear cached STS credentials on Init to avoid reusing credentials obtained for a previous storage configuration
@rufusu
rufusu requested a balanced review from Copilot September 17, 2026 17:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical token and object-path issues, plus multipart upload edge cases, remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (3)

drivers/rakuten_drive/driver.go:357

  • The configured part size is only clamped to the 5 MiB minimum, so a file larger than roughly 50 GiB at the default size produces more than S3's 10,000-part limit and the upload fails. Please either increase the part size dynamically to stay within s3.MaxUploadParts or reject oversized uploads before creating the multipart upload.
	partSize := d.UploadPartSize
	if partSize < 5*1024*1024 {
		partSize = 5 * 1024 * 1024
	}
	parts, err := d.uploadMultipart(ctx, s3Client, initResp.Bucket, objectKey, s3UploadID, reader, size, partSize, up)

drivers/rakuten_drive/driver.go:545

  • An empty file reaches uploadMultipart with size == 0, which immediately returns file size required for multipart upload. Empty files are valid objects, so every zero-byte upload fails after the upload-init call; handle this case with a non-multipart/zero-byte upload flow (and the corresponding completion request) instead of rejecting it here.
	if size <= 0 {
		return nil, fmt.Errorf("file size required for multipart upload")

drivers/rakuten_drive/util.go:145

  • When the API reports credentials that expire within the next minute, this branch replaces their real expiration with a fresh 30-minute TTL. The cached STS credentials can therefore be reused after they have expired, causing subsequent multipart uploads to fail until the cache is reset; only an unknown expiration should use the fallback, while near-expiry credentials should not be cached (or should be rejected/refetched).
	expire := parseTimeAny(tokenResp.Expiration)
	if expire.IsZero() || !time.Now().Before(expire.Add(-time.Minute)) {
		// unknown or nearly-expired credential; use a conservative TTL
		expire = time.Now().Add(30 * time.Minute)
  • Files reviewed: 6/6 changed files
  • Comments generated: 2
  • Review effort level: Lite

Comment thread drivers/rakuten_drive/driver.go
Comment thread drivers/rakuten_drive/driver.go Outdated
…nd STS expiry

- Scale the part size up when needed so large uploads stay within the S3 10,000-part limit instead of failing
- Upload empty objects with a single PutObject request, since S3 multipart upload requires at least one part and every zero-byte upload previously failed
- Stop extending nearly-expired STS credentials with a 30-minute cache TTL; return them for the current call without caching so the next request refetches

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Four moderate issues remain in upload path handling, overwrite behavior, part-size validation, and token synchronization.

Review details

Suppressed comments (4)

drivers/rakuten_drive/driver.go:415

  • The returned object ID is only initResp.File[0].Path, while listing normalizes each item against remoteDir. For uploads into a nested directory this leaves the object ID without its parent path, so the immediately returned object is later passed to Link, Move, Rename, or Remove with an incomplete remote path. Normalize this path against remoteDir before storing it as the ID, consistently with parseList.
			ID:       initResp.File[0].Path,

drivers/rakuten_drive/driver.go:314

  • OpenList's upload flow treats an existing destination as overwrite unless NoOverwriteUpload is enabled (see internal/op/fs.go:635-643), but this request hardcodes replace to false. As a result, uploading a file over an existing non-empty Rakuten file will be rejected or handled as a conflict instead of honoring the storage's normal overwrite behavior; pass the overwrite setting expected by the API (or explicitly derive it from the driver's configuration).
	initBody := base.Json{
		"host_id":   d.HostID,
		"path":      remoteDir,
		"file":      []base.Json{{"path": file.GetName(), "size": size}},
		"upload_id": "",
		"replace":   false,

drivers/rakuten_drive/driver.go:371

  • UploadPartSize is only clamped to the 5 MiB minimum here. A configured value above S3's 5 GiB maximum is accepted and then passed to make([]byte, partSize) in uploadMultipart, so a bad configuration can either allocate an enormous buffer or cause S3 to reject every part. Clamp the value to the service maximum (and preferably validate it during initialization) before allocating the buffer.
		partSize := d.UploadPartSize
		if partSize < 5*1024*1024 {
			partSize = 5 * 1024 * 1024
		}

drivers/rakuten_drive/util.go:114

  • doForestRequest reads d.accessToken without the mutex, while ensureAccessToken and invalidateAccessToken write it under tokenMu. Concurrent requests can therefore race (and the race detector can report this), especially when one request receives a 401 and invalidates the token while another is being sent. Take a mutex-protected snapshot of the token before constructing the request; the existing 401 retry will handle a stale snapshot.
	req := d.client.R().
		SetContext(ctx).
		SetHeader("Authorization", "Bearer "+d.accessToken).
  • Files reviewed: 6/6 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

…token race

- Normalize the object ID returned by Put against the upload directory so uploads into nested directories keep their parent path, consistently with parseList
- Derive the upload replace flag from the op layer's exist flag instead of hardcoding false, honoring OpenList's default overwrite semantics
- Clamp the configured part size to the S3 5 GiB maximum in addition to the 5 MiB minimum so oversized configurations cannot allocate huge buffers
- Take a mutex-protected snapshot of the access token in doForestRequest to avoid racing with invalidation from the 401 retry path

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Address the multipart upload memory risk and the identified listing and token reinitialization issues.

Get a fresh assessment by requesting another Copilot review.

Review details

Suppressed comments (2)

Previously missed (1) — in code that hasn't changed since the last review.

drivers/rakuten_drive/driver.go:469

  • The {"data"} candidate is selected before the nested {"data", "file"}/files/items candidates whenever data is an object, because Size() is non-zero for object fields. For a response shaped like {"data":{"files":[...]}}, this makes the parser iterate over the object's values instead of file entries and can return malformed or empty listings. Try the nested data paths before the bare data path, or only accept data directly when it is an array.

drivers/rakuten_drive/driver.go:57

  • Init clears only the STS cache, so reinitializing this driver after changing RefreshToken or HostID can keep the previous unexpired accessToken and authenticate the new configuration with stale credentials. Clear accessToken and accessTokenExpire under tokenMu here as well before calling ensureAccessToken.
	// drop credentials cached for a previous configuration
	d.stsMu.Lock()
	d.stsCreds = nil
	d.stsMu.Unlock()
  • Files reviewed: 6/6 changed files
  • Comments generated: 1
  • Review effort level: Lite

Comment thread drivers/rakuten_drive/driver.go Outdated
Comment on lines +373 to +375
if partSize > 5*1024*1024*1024 {
partSize = 5 * 1024 * 1024 * 1024
}
…t sizing

- Only accept array-shaped candidates when locating the entry list in parseList, and try the nested data paths before the bare data path, so a response like data/files is not iterated as the object's fields
- Clear the cached access token and its expiry in Init as well, so reinitialization after a config change cannot authenticate with the previous configuration's token
- Bound the configured part size to 1 GiB in addition to the 5 MiB minimum so a misconfigured value cannot allocate an enormous in-memory buffer
- Add parseList regression tests covering the data-object shape, flat file arrays, and responses without a list field
@chatgpt-codex-connector

Copy link
Copy Markdown

To use Codex here, create a Codex account and connect to github.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Critical upload memory handling and additional timestamp, path, and credential issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity

Open (2)
Resolved since last review (1)

Comment on lines +384 to +386
if maxParts := int64(s3MaxUploadParts); (size+partSize-1)/partSize > maxParts {
partSize = (size + maxParts - 1) / maxParts
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement Module: Driver Driver-Related Issue/PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants