Expose Encoder and Decoder in TiktokenTokenizer

[razshare]

Fixes #7313

We are excited to review your PR.

So we can do the best job, please check:

• There's a descriptive title that will make sense to other developers some time from now.
• There's associated issues. All PR's should have issue(s) associated - unless a trivial self-evident change such as fixing a typo. You can use the format Fixes #nnnn in your description to cause GitHub to automatically close the issue(s) when your PR is merged.
• Your change description explains what the change does, why you chose your approach, and anything else that reviewers should know.
• You have included any necessary tests in the same PR.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.10%. Comparing base (5090327) to head (140aa42).
Report is 44 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7314      +/-   ##
==========================================
+ Coverage   68.87%   69.10%   +0.23%     
==========================================
  Files        1470     1475       +5     
  Lines      274005   277261    +3256     
  Branches    28403    29043     +640     
==========================================
+ Hits       188717   191606    +2889     
- Misses      77970    78374     +404     
+ Partials     7318     7281      -37     

Flag	Coverage Δ
Debug	62.64% <100.00%> (+0.01%)	⬆️
production	62.64% <100.00%> (+0.01%)	⬆️
test	∅ <100.00%> (∅)

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
...Microsoft.ML.Tokenizers/Model/TiktokenTokenizer.cs	78.28% <100.00%> (ø)
...est/Microsoft.ML.Tokenizers.Tests/TiktokenTests.cs	99.39% <100.00%> (+0.40%)	⬆️

... and 56 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[tarekgh]

@razshare I replied on the issue. Let's discuss it there first before we continue here. Thanks a lot for your submission. I converted this PR to be draft for now till we finish the discussion.

[razshare]

@dotnet-policy-service agree

[dotnet-policy-service]

Draft Pull Request was automatically closed for 30 days of inactivity. Please let us know if you'd like to reopen it.