this is continous action space since action is a real number or discrete as number of elements (100 here ) are fixed ?
I need an algorithm that combines both discrete and continuous actions space, like if I wanted to choose action but I needed to also perform that action at an optimized velocity or something. Can I do this with ppo and how. I am still a beginner but must of the algorithms I played with had either separately I thought of actor cri but I don’t think the value network does what I want I can’t use that to predict how much velocity to use
Hello there,
I noticed that in many works by OpenAI and Deepmind in RL, they deal with a continuous action space which they discretize. For example rather than having to continuous actions to determine speed and direction of movement, they would discretize speed and torque.
Is there any reason why discretized action space is more favorable?
I am trying to implement RL using OpenAI with Stablebaselines3 to train a model for a real-life experiment. Its discrete action space consist of ~250 different possible actions. (action consists of combination of two different discrete actions. Example of action are 1, 5 or 10, 25) and a continuous state space (sensor readings).
I find that DQN might fit, as internet says it is: sample efficient, works for discrete action & continuous state space. Would it have trouble learning with such many different of actions?
Also, from my expert knowledge of the environment, actions with similar value would impact the result similarly. For example, given identical state, resulting reward of an action of (3, 15) will not be drastically different from that of the (4, 14). Would DQN be able to quickly generalize?
Hi everyone,
I'm stuck on this metric spaces question. Any help is greatly appreciated:
Let X be a non-empty set and let d0 be the discrete metric on X. Under what circumstances is (X, d0) connected?
Thought process so far:
I have that it's only connected if X has 1 element and I can see why that is (AUB would not have A and B as disjoint, non-empty sets), but I don't know how to disprove a larger set X.
Thanks in advance!
Hello all ..I am quite new to RL and even ML. So I would like to ask a very basic question.
My problem has a continuous observation space (ranges from 30 to 200) and my action space is discrete (move a point up or down by a fixed amount) . I want my agent to select a correct action based on the observation. Say for example I always want my observed value to reach the value 60, and say this will be achieved if i moved the point to a particular value which will make my observation 60.
How do put such a logic to implementation?
Hi,
I am using a policy gradient approach to do a graph optimisation problem. I have been flitting back and forth between different solution methods to the large action space in the environment. At each step, the agent can remove an edge between two nodes or add an edge between two nodes. The graph itself is quite large (~10k nodes).
To overcome the large action space i have tried limiting the number of nodes which has worked, however I want to also now start including all nodes in the environment by splitting the large action space in to smaller sub actions. Conceptually this is fine, however, I am unsure as to how to implement this specifically in the context of the MDP.
The sub actions would consist of 3 actions.
- action 1: Choose node 1 out of 10k nodes (action space = 10k)
- action 2: Choose node 2 out of 10k nodes (action space = 10k)
- action 3: if edge exists remove, if edge does not exist add.
This makes the problem tractable even though learning will take a long time (if it can learn at all) - this is besides the point though - I am unsure about how to implement this and how this fits in to the MDP.
Is it possible to define three separate actions in the environment to reflect the three actions above and then use a neural network say with 3 heads each with the number of actions... to determine how good the actions selected were and increase/decrease the probability of choosing those combinations? If so, how is head1 mapped to action space 1, head2 with action space 2 etc?
Is there an example implementation of something like this? thank you!
There is a proof in a relatively famous paper in psycholinguistics (Levy, 2008) that invokes a KL divergence between discrete distributions p and q that do not seem to be defined over exactly the same space (p's domain is a proper subset of q's). Am I misunderstanding the proof, or is it wrong?
For context, Levy is considering a situation where people have probability distributions defined over linguistic constructions, where each construction is consistent with exactly one sequence of words (but the reverse is not true: one word sequence is consistent with many constructions). Every time a word is encountered, people update this distribution over possible constructions. The contribution of his paper was to show that the KL divergence (distribution after seeing the word || distribution before seeing the word) is equivalent to the surprisal of the word under the old distribution. This is his proof, and this is my problem with it.
Hi,
I am studying the problem of the differentiability of maps (functions) for which only some of the input are topological spaces.
I have a graph G={V,E} comprising a set of vertices V and a set of edges E. Disregard the set E for this question. There is also a 2-D metric space X, and all vertices in V are positioned somewhere in X.
There is a set of features or attributes A={a_1, a_2, ..., a_{|V|} } which correspond to each of the vertices in V. Each a_n is a vector, some components are taken from discrete sets, some are taken from continuous sets.
I have a map of the form F: (X,V,A) \to R. Questions:
- Is F a differentiable function? The answer should be yes, because X is continuous and therefore there should be at least one partial derivative dF/dX.
- Is F differentiable over A? Only some of the features of A are continuous, while the others are discrete (boolean, for example). I do not know how to answer this question. If F is not generally differentiable over A, how do I indicate that it is differentiable over the features of a_n (for example, the i-th component of a_n) that are continuous?
- Is F differentiable over V? The answer to this question should be no, I have included it here for consistency check.
Does anybody know some good recent survey papers about the current state of RL algorithms for discrete action spaces? I am currently in the literature review phase of my master thesis, and I already know that my RL implementation will have a discrete action space. I am unsure about how large the action space will be and what the RL agent will do exactly. That is why I want some more knowledge about all the different RL algorithms, such that I can design my RL implementation with the strengths and limitations in mind of the RL algorithm that I will use. For instance, I would love to read some papers that can tell me which RL algorithm works the best with a small action space, large action space or even if the size of the action space can alter between problems.
Hopefully, someone on this subreddit knows where I should start looking. Other survey papers, which for instance, compare different DQN implementations or Policy based algorithms, are also more than welcome.
Hello there, this is the continuation of my previous post.
After some investigation, I found that the problem is not with my MPC controller but with the discrete state space model.
I discretized my quadcopter's continuous state space model, which has good outputs on all the 10 out of 12 states. Height and rising velocity of the quadcopter being the 11th and 12th ones which can't be controlled. And yes my model is both controllable and observable.
I found that even with zero thrust force, the discrete model can rise up in height.
Can anybody suggest any ideas on how to tackle it?
Hi all!
I am currently working on a Q-Learning project with a discrete action space. Two agents can choose from a set of, for example, ten actions. Both agents learn independently and have no means of communication.
Now my goal is to add a shock (in the form of random noise) after a random time interval so that the action space changes and affects the future reward of the two agents so that the behaviour may change.
My question to you would be, how can I implement this most effectively? My goal would be to have the action space change with some probability after a time so that the agents have to adapt to the new circumstances.
I'm currently practicing my LQR controller on a discretised state space model of a cart pole problem.
I first hand calculated the continuous SS model, used c2d (matlab) to discretised it for a sample time of Ts (0.01).
I then created a close-loop system called sys_dsc_cl using ss(AA-BB*KK, BB, CC, DD, Ts)
I did a step() and it shows that the 4 states are able to stablise as expected. But the puzzling thing is that when I queried the eigenvalues, It is clear that the real parts of all 4 eigenvalues are positive, which is contradicting whatever that I have learnt that eigenvalues (poles) need to be negative in order to be stable.
mp = 0.3;
mc = 0.5;
L = 0.15;
g = 9.81;
A = [0 1 0 0; 0 0 (mp*g/mc) 0; 0 0 0 1; 0 0 ((mp+mc)*g/(L*mc)) 0];
B = [0; 1/mc; 0; 1/(L*mc)];
C = eye(4);
D = [0; 0; 0; 0];
Q = diag([1, 1, 10, 100]);
R = 1;
freq = 100;
Ts = 1 / freq;
sys_cont = ss(A, B, C, D);
sys_dsc = c2d(sys_cont, Ts);
[AA, BB, CC, DD] = ssdata(sys_dsc);
KK = dlqr(AA, BB, Q, R);
sys_dsc_cl = ss(AA-BB*KK, BB, CC, DD, Ts);
% step(AA-BB*KK, BB, CC, DD)
eig(sys_dsc_cl)'
eig(AA-BB*KK)'
step(sys_dsc_cl)
output:
0.2840 + 0.0000i 0.9930 - 0.0074i 0.9930 + 0.0074i 0.9907 + 0.0000i
0.2840 + 0.0000i 0.9930 - 0.0074i 0.9930 + 0.0074i 0.9907 + 0.0000i
https://preview.redd.it/yl4gc32b1rt61.jpg?width=560&format=pjpg&auto=webp&s=6410bd17a9829ee87ded9719c5bc5346a7ad03b3
I even verified inside simulink by doing the below. Note that inside the discrete state-space block, the values are AA, BB, CC, DD and an initial state of [0 0 deg2rad(5) 0].
https://preview.redd.it/xx82oevq1rt61.jpg?width=1201&format=pjpg&auto=webp&s=a58dfe5b7194abc45325b05ca93c62de8e246756
https://preview.redd.it/nj2r9e8y1rt61.jpg?width=3000&format=pjpg&auto=webp&s=f3f4667488346252293589c021e5554669ff9059
I looked at the scope and it shows it stabilising (similar to the step function in matlab).
All in all, everything stabilises, even though their eigenvalues are positive! Can someone tell me where I got it wrong?
I'm pretty sure the stabilising part is correct, but for some reason, the extraction of the eigenvalues (poles) are getting wrong.
Can someone advise where I went wrong please?
Hi, im fairly new to RL and as my first own project i wanted to make a cool snake implementation with RL. However, after trying it with Q learning, the results were not as good as I expected. So i went on to read about TD3 and SAC, however they are both intended for continuous action spaces. Is there any similar algo that i can use for discrete AS? Ive read about discrete SAC, is it any good? Or can i even simply "discretize" the output of SAC/TD3 so that i assign every interval a certain action? Answers, explanations and papers would be very appreciated.
https://i.redd.it/thvnxea9lao61.gif
So this is more of just a fun post: I'm curious if anyone had any applications or ideas which used fractional operators . I also wanted to show of my gif.
A bit of background on what's in the GIF and fractional operators in general.
Recall that the 2D discrete Fourier transform, F, is a linear operator on a space of matrices (or d by d arrays). If we apply the Fourier transform 4 times it returns the identity function i.e. F^4 = F(F(F(F))) = I. Note that people have figured out how to let these exponents take non-integer values! This corresponds to fractional Fourier transforms. So for example the half Fourier transform F^(1/2) is something that functions like the square root of the Fourier transform. If we let G = F^(1/2) then we have that G(G)=F, or maybe a bit more concretely, for any matrix/image X, we have that G(G(X)) = F(X). These special exponents behave like regular old exponents in a lot of ways and it has been observed that one can construct F^a for arbitrary real-valued a.
The GIF I've posed takes an image of a pagoda X and applies increasing fractional degrees of Fourier transforms. Specifically the graph shows shows F^a (X) as a goes from 0 to 4.
Links, more on fractional operators
- Fractional fourier transform: https://en.wikipedia.org/wiki/Fractional_Fourier_transform
- It worth nothing that its possible to take the square roots of many linear transforms thereby making it possible to construct fractional versions of these transforms in a straightforward manner. https://en.wikipedia.org/wiki/Square_root_of_a_matrix
- If you think this is cool you should also check out fractional calculus: https://en.wikipedia.org/wiki/Fractional_calculus
(Remember, the derivative is also a linear operator 🙂....)
Conclusion
I'm curious if anyone has any interesting ideas...
Recently, I ran into a few papers that re-formulate Soft Actor Critic for a discrete action space on the premise that the original algorithm is incompatible with discrete action spaces:
I don't understand why this work is necessary. It seems to me that changing the output of the policy network in the usual way (output a scalar for each action and apply a softmax) would be 100% valid using the original description of the algorithm. The only benefit I see is the possibility of replacing some Monte-Carlo estimates in the loss functions for the value/action-value networks.
Am I missing some detail that breaks the original SAC algorithm in the discrete action setting?
I'm new to RL techniques, so this may be a dumb question.
It is a combinatorial scenario. For example, the action space is 5*3*4, i.e., I have to make a choice in each of the 3 sub-action sets, which as a whole constitutes the action. I hope the neural network can directly output the 3 sub-actions, perhaps by 3 neurons in the output layer.
I was reading through this post and was curious about some of the same things as the OP.
One of the main reasons I wanted to move away from DQN was that I liked the idea of optimizing directly on the policy instead of trying to predict Q values.
For my particular problem, I don't really care about predicting a specific return value, I really only care about knowing that action A is better than action B or visa versa. My main concern is that I am afraid a lot of my model's capacity will be used trying to predict specific values when value predictions are very difficult and/or are stochastic in nature. Which action is better is lot less random and more tractable of a problem to solve.
In addition to that I think that loss from predicting Q-distributions does not necessarily translate to higher reward in a direct way. For example a 50% decrease in loss predicting Q distributions might not equate to a 50% increase in reward, or even anything close. Obviously they will be generally correlated such that lower loss will eventually translate to higher reward but I am worried the model will spend capacity to reduce loss in ways that don't really translate well to higher overall reward.
That being said it seems like a lot of policy gradient type methods like actor-critic or PPO do actually still need a value function. A lot of the responses in the post above seem to indicate there isn't much of a benefit to using actor-critic on discrete action spaces. We still have a value function so my concerns above still stand. Is my understanding correct in that gradient-methods still rely on predicting Q-values or is this not the case?
For example, if we know the xy coordinates are (10cos(K*pi) , 10sin(K*pi)) for time K, issit actually possible to develop a state space model to expres x(k+1) and y(k+1) in terms of x(k) and y(k)?
It seems like it should be so easy but yet, I am totally stumped. Im trying to form a state space model to eventually apply Kalman filter to remove noisy measurements, but I cant even form the State Space model
It would be great if someone could point me to some gym-type environments which have discrete action space and for which, making a handcrafted controller would be easy. One environment which I could find was the LunarLander-v2 where they have already defined a heuristic-based controller. Are there other environments where I could possibly have such controllers?
Also, it would be great if someone could suggest possible ideas for designing an environment for a particular task with the above specs from scratch if it doesn't exist (any domain is fine: robotics, games, trading, etc.). I wanted to make heuristic-based controllers as a baseline to compare it against the performance of an RL-agent.
I have experience using stable baselines, but they cannot provide Tuple/Dict action spaces. Do you have an idea of a framework similar to stable baselines that I can use tuple action spaces?
I am currently struggling with DQN in the case of multi discrete action spaces. I know that the output layer of the Deep Q Net should have the same dimensionality of the discrete action space. What if I have several discrete action spaces because I want my Q agent to perform simultaneously different actions? How does the architecture of the net change?
I figured out that the output layer should have the dimesion of all possible combinations of the discrete action spaces, but in that case the dimensionality of the output layer would grow fast. Moreover, in doing that each neuron in the output layer would be associated with N values instead just one, where N is the number of simultaneous actions that can perform.
Is there a more efficient and easier way to deal with it?
[paper here] The idea is to predict the action is continuous space and then the environment maps it to k valid discrete reactants using k-NN. This enables them to use any continuous action space RL algorithm and the results are pretty impressive
I'm using RL framework to address a problem where my action is defined in a n-dimensional discrete space where n=3. For each action we can have a integer value between 1 and 50 (e.g. one action could be At = {'a1': 3, 'a2': 45, 'a3': 23}).
As I'm using Actor-Critic I'm wondering about the actor network and more precisely what should it learn. Since using all possible combination with order and repetition (and thus 50^3 different actions) would force the neural network to handle a 125000 output layer shape, I'm quite unwilling to go this way. I thought about switching to 3-dimensional continuous action space where I could learn the mean and std of a Gaussian distributions that I would use to sample action from (and round the value). On the other hand this approach would assume that it exists a Gaussian dependency within each action dimension which is (I guess?) not the case in my project. (The Poisson distribution would have the same false(?) assumption) So my question is: How this kind of setting (multidimensional large discrete action space) can be addressed ?
Help is very welcomed, thank you very much.
Can anyone help me solve this? Or just solve it for me, as i won't understand anything :D
I've seen a few actor-critic methods for discrete action spaces, however I do not seem to understand how they work.
If I have a discrete action space, isn't an actor-critic architecture equivalent to standard Q-learning where I use a softmax over the Q-values to create my policy? Why do I need to two networks?
For continuous action spaces it makes sense to me, since I can't do a max over the action values, so instead I can use another network to evaluate my action. So I guess my question is, does actor-critic architectures only make sense for continuous action spaces?
Thanks!
I have an assignment to design a Discrete state space controller for a continuous plant in Simulink. I have written the Matlab code to calculate the controller vector and what happens is that as long as the system is continuous the Matlab and the Simulink step responses are identical.
After transitioning to discrete time the Matlab step response is still identical to the continuous step response (albeit it is now discrete as expected), but I just cannot get the discrete Simulink modell to work.
Matlab code:
%state space model of DC motor
J=1.05e-6;
R=7.73;
L=0.000832;
K=0.044;
b=(K^2*0.0144)/(24-R*0.0144);
A=[0 1 0 ; 0 -b/J K/J ; 0 -K/L -R/L];
B = [0 ; 0 ; 1/L];
C = [1 0 0];
D = [0];
motor_ss = ss(A,B,C,D);
%contiuous state feedback w/ integrator
poles_i=[-100-1i -100+1i -200 -500]; %poles in s-domain
Ka=acker([A zeros(3,1); -C 0],[B;0],poles_i);
Kx=Ka(1:3);
Ki=Ka(4);
sys_cli=ss([A zeros(3,1); -C 0]-[B;0]*Ka,[0;0;0;1],[C 0], D);
%convert to discrete state-space
Ts=1/1000;
motor_d=c2d(motor_ss,Ts,'zoh');
A_d=motor_d.a;
B_d=motor_d.b;
C_d=motor_d.c;
D_d=motor_d.d;
%create discrete controller w/ integrator
poles_d=exp(poles_i.*Ts); %poles in z-domain
Ka_d=acker([A_d zeros(3,1); -C_d 0],[B_d;0],poles_d);
Kx_d=Ka_d(1:3);
Ki_d=Ka_d(4);
sys_cld = ss([A_d zeros(3,1); -C_d 0]-[B_d;0]*Ka_d,[0;0;0;1],[C_d 0], D_d, Ts);
%precompensation
Nbar_d=1/dcgain(sys_cld);
sys_cld = ss([A_d zeros(3,1); -C_d 0]-[B_d;0]*Ka_d,[0;0;0;1]*Nbar_d,[C_d 0], D_d, Ts);
step(sys_cli)
step(sys_cld)
grid on
stepinfo(sys_cli)
stepinfo(sys_cld)
EDIT: I have spent a considerable ammount of time trying to figure out why this doesn't work, any ideas to point me towards a solution would be much appreciated.
EDIT2: the plant has to be continuous
EDIT3: Simulink files: continuous and [discrete](https://hszkbmehu-my.sharepoint.com/:u:/g/personal/bd1111_hszk_bme_hu/EbUO2pzHrJl
... keep reading on reddit ➡Hey!
Im looking for a sff case for a i7 6700k (non oc´d) and a sfx psu. There will be no discrete gpu, only a pcie wifi card (Fenvi t919 for hackintosh, could theoretically go into the m.2. wifi/bt slot but the card has 4 antennas...).
There are a few nice cases which could hold theses components but the problem seems to be the cooling. I tried the Noctua l9i as the most recommended low profile cooler, but the cpu got way too hot and the cooler too loud. Cooling with a big shuriken v2 rev.b was okay, so 58mm cooler height should be enough. Currently the build house inside a phantek evolve shift with an h80iv2 but it is way too big for such a small build. So are there any recommendations, best below 15l and about ~100€?
Cases I already looked at:
DanCase, Ghost, etc... -> Too expensive for this build and a low of waisted space because of the missing gpu
Inwin Chipin -> Don't think there is a good cooling option for the 6700k
Silverstone Milo, Node202 -> Again, too much wasted gpu space and I personally don't like the formfactor
Thanks in advance :)
