fix(http-client-csharp): preserve explicit null for optional nullable properties

[JoshLove-msft]

Summary

Fixes #11970.

Generated model classes now distinguish an untouched optional nullable property from one explicitly assigned null. Omitted properties remain omitted, explicit null writes JSON null, and non-null values keep their existing representation. JSON deserialization preserves these states for subsequent JSON and wire-format serialization.

The generator uses private backing storage and nonpublic presence flags for scalar/model properties, and reuses change-tracking collection state for nullable lists and dictionaries. It preserves inherited properties, including dynamic models with a non-dynamic base, without changing public property types or constructor/factory signatures. Expected projects were regenerated with the supported scripts; generated files were not hand-edited.

Compatibility boundaries

• Required-nullable, optional-nonnullable, read-only wire filtering, XML serialization, and JsonPatch override precedence retain their existing behavior.
• Defaulted constructor/factory scalar arguments retain null-as-omitted semantics. Existing factory collection initialization also remains unchanged, including default lists materialized as empty lists. Assign a writable property after construction to request explicit null.
• Handwritten replacement properties/fields and models customized as readonly structs retain their existing behavior; no public presence-tracking API or replacement-member opt-in is introduced. These boundaries are documented in the customization guide.

Coverage and validation

Regression coverage includes omitted/null/value states, assignment transitions, model references, strings, integers, booleans, enums, dates, bytes, nullable collection elements, nested wire serialization, both JSON interfaces, round trips, duplicate-property last-value-wins, inheritance, name collisions, handwritten customization, factory defaults, read-only properties, and dynamic patches.

• 4,538 generator/local unit tests passed.
• 31 real Spector nullable and JSON merge-patch tests passed; test libraries were regenerated back to their standard stub form afterward.
• 244 emitter tests passed, with 2 existing skips (--maxWorkers=4 avoids contention against the existing five-second timeout).
• Full Generate.ps1, focused final regeneration, npm run build, API extraction, repository/emitter formatting, and C# formatting passed.
• Cop checks passed. Repository oxlint scanned 70 emitter files with zero diagnostics; the package's existing npm run lint still references removed ESLint tooling, so the repository's current linter was used instead.

No C# changelog/Chronus entry is included.

[pkg-pr-new]

Open in StackBlitz

npm i https://pkg.pr.new/@typespec/http-client-csharp@11971

commit: e6f7fbd

[github-actions]

No changes needing a change description found.

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Adds presence tracking so C# models preserve omitted, explicit null, and non-null states during JSON and wire serialization.

Changes:

• Adds backing fields and presence flags for nullable scalar/model properties.
• Preserves nullable collection and dynamic-model states.
• Adds regenerated samples, regression tests, and customization documentation.

File summaries

File	Description
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/SampleTypeSpecModelFactory.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/Thing.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/Thing.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/SampleTypeSpecContext.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableProperties.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableProperties.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableDynamicProperties.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableDynamicProperties.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableContainer.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableContainer.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableChild.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableChild.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableBase.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/OptionalNullableBase.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/NullableDynamicModel.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/NullableDynamicModel.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/src/Generated/Models/DynamicModel.Serialization.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local/Sample-TypeSpec/Sample-TypeSpec.tsp	Updated as part of this pull request.
packages/http-client-csharp/generator/TestProjects/Local.Tests/ModelSerializationExtensionsTests.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/ScmModelProvider/TestData/ScmModelProviderTests/OptionalNullableCustomPropertyRetainsHandwrittenBehavior/Model.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/ScmModelProvider/ScmModelProviderTests.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanReplaceSerializationMethod.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanReplaceDeserializationMethod.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeSerializationMethod.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeDeserializationMethodWithoutOptions.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/SerializationCustomizationTests/CanCustomizeDeserializationMethodWithOptions.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/test/Providers/MrwSerializationTypeDefinitions/TestData/MrwSerializationTypeDefinitionTests/SerializedNameIsUsed(False).cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/MrwSerializationTypeDefinition.Xml.cs	Updated as part of this pull request.
packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/MrwSerializationTypeDefinition.cs	Updated as part of this pull request.
packages/http-client-csharp/.tspd/docs/customization.md	Updated as part of this pull request.

Review details

Suppressed comments (2)

packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs:209

• This treats every redeclared inherited property as sharing the base presence flag, even when ModelProvider marks the derived member new for a domain change (ModelProvider.cs:704-719). The generated new property keeps its field-based setter because this update only rewrites Override setters, so assigning null or a value never sets the shared flag and serialization omits the property; share the flag only for overrides and let new properties allocate their own state.

            if (property.BaseProperty != null)
            {
                var baseProperty = BaseModelProvider?.Properties.FirstOrDefault(p => p.Name == property.BaseProperty.Name);
                if (baseProperty != null && GetNullablePropertyPresence(baseProperty) is { } basePresence)
                {
                    _nullablePropertyPresence[property] = basePresence;
                    if (property.Modifiers.HasFlag(MethodSignatureModifiers.Override))
                    {
                        property.Update(body: new MethodPropertyBody(
                            Return(Base.Property(baseProperty.Name)),
                            property.Body.HasSetter ? Base.Property(baseProperty.Name).Assign(Value).Terminate() : null));
                    }
                }
                return;

packages/http-client-csharp/generator/Microsoft.TypeSpec.Generator.ClientModel/src/Providers/ScmModelProvider.cs:173

• This reservation only includes generated m.Fields, but inherited handwritten fields are kept in m.CustomCodeView.Fields and merged separately by CanonicalTypeProvider (Microsoft.TypeSpec.Generator/src/Providers/CanonicalTypeProvider.cs:246-247,305-306). A base customization with an accessible field such as _textIsDefined can therefore receive the same name as a generated presence field in a derived model, producing a field-hiding warning; this generator treats warnings as errors (generator/Directory.Build.props:29). Include inherited CustomCodeView.Fields in the reserved names and add an inheritance/custom-field regression test.

            return ancestors.SelectMany(m => m.Fields.Select(f => f.Name));

• Files reviewed: 14/32 changed files
• Comments generated: 2
• Review effort level: Lite

---

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

🟡 Changes recommended

Nullable collection properties lose explicit null presence and serialize as omitted.

Get a fresh assessment by requesting another Copilot review.

Review details

• Files reviewed: 17/37 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Copilot review overview

🟡 Changes recommended

Boolean assertions need correction, and focused XML presence coverage is still missing.

Review effort: Lite
Findings: 1 · 1

Open (2)

• Nullable collections are still presence-tracked only through Optional.IsCollectionDefined, which…
• Add XML regression coverage for preserveJsonPresence false · New

[Copilot]

Copilot review overview

🟡 Changes recommended

The critical XML null-collection issue and moderate custom-base field collision remain unresolved.

Review effort: Lite
Findings: 1

Open (1)

• XML serialization throws for null optional collections · New

Resolved since last review (2)

• Nullable collections are still presence-tracked only through Optional.IsCollectionDefined, which…
• Add XML regression coverage for preserveJsonPresence false

[Copilot]

Copilot review overview

🟡 Changes recommended

Nullable additional properties currently lose explicit null values during deserialization and subsequent wire serialization.

Review effort: Lite
Findings: 1

Open (1)

• Null additional properties are dropped during deserialization · New

Resolved since last review (1)

• XML serialization throws for null optional collections

[Copilot]

Copilot review overview

🔵 Needs a closer look

The broad generator, serialization, XML, inheritance, dynamic-model, and regenerated-fixture changes require final human review.

Review effort: Lite
Findings: None

Resolved since last review (1)

• Null additional properties are dropped during deserialization