Stop Exploring all State Space for RL !

Invariant Subspaces on Quantum Circuits for Reinforcement Learning

An invariant subspace provides a way to restrict a quantum RL agent to a smaller part of its Hilbert space while ensuring that the quantum dynamics remain inside that part of the space.

Get Farshad Noravesh’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Algorithm: Quantum Reinforcement Learning with Invariant Subspace Reduction

Input:
    Environment E
    Number of qubits n
    Subspace dimension k << 2^n
    Learning rate η
    Number of episodes N
    Maximum episode length T

--------------------------------------------------
1. Construct the invariant subspace
--------------------------------------------------

H ← 2^n-dimensional Hilbert space

Choose a set of basis states/eigenvectors:
    {|φ1>, |φ2>, ..., |φk>}

Construct the subspace:
    S = span{|φ1>, ..., |φk>}

Construct the projector:
    P_S = Σ_i |φi><φi|

Verify that the policy dynamics preserve S:
    Uθ S ⊆ S

--------------------------------------------------
2. Initialize quantum policy
--------------------------------------------------

Initialize trainable parameters:
    θ

Construct a parameterized quantum circuit:
    Uθ

Ensure that Uθ preserves the invariant subspace.

--------------------------------------------------
3. Quantum state encoding
--------------------------------------------------

For each environment state s:

    Encode s into the reduced subspace:

        |ψ_s> = V_enc(s)|φ0>

    where |ψ_s> ∈ S

--------------------------------------------------
4. Generate quantum policy
--------------------------------------------------

    |ψ'_s> = Uθ |ψ_s>

Because S is invariant:

    |ψ'_s> ∈ S

--------------------------------------------------
5. Convert quantum state to action probabilities
--------------------------------------------------

    p(a | s, θ) = Measure(|ψ'_s>)

Select action:

    a ~ p(a | s, θ)

--------------------------------------------------
6. Interact with environment
--------------------------------------------------

    Execute action a

    Observe:
        reward r
        next state s'

    Store:
        (s, a, r, s')

--------------------------------------------------
7. Update quantum policy
--------------------------------------------------

Calculate return:

    G_t = Σ_{j=t}^{T} γ^(j-t) r_j

Calculate policy gradient:

    ∇θ J(θ)
      = Σ_t G_t ∇θ log p(a_t | s_t, θ)

Update parameters:

    θ ← θ + η ∇θ J(θ)

--------------------------------------------------
8. Repeat
--------------------------------------------------

Repeat steps 3–7 for N episodes.

Return:
    Optimized quantum policy Uθ*
    Invariant subspace S

Environment State s
       │
       ▼
 Classical State Encoder
       │
       ▼
 Reduced Quantum State |ψs> ∈ S
       │
       ▼
 ┌─────────────────────┐
 │ Parameterized        │
 │ Quantum Policy Uθ    │
 │                      │
 │ Uθ S ⊆ S             │
 └─────────────────────┘
       │
       ▼
 Measurement
       │
       ▼
 Action Distribution πθ(a|s)
       │
       ▼
 Environment
       │
       ▼
 Reward
       │
       └──────────► Policy Gradient
                         │
                         ▼
                         θ update